• Resolved glashsix

    (@glashsix)


    Hi just wanted to give a heads up.

    After doing something today, that you should never do! ( learned this the hard way )… I updated the plugin to the new version in the morning in production, and ended up breaking things pretty badly.

    The new update changed something that caused our server to become unresponsive. Couldn’t really investigate what and why, but php-fpm didn’t function properly after the update.

    We removed the plugin completely until we are sure it’s going to be fine to put it back.

    Good Luck

Viewing 15 replies - 1 through 15 (of 27 total)
  • Hello @glashsix

    We’re sorry you’re facing that issue, we would love to help you more so could you turn on Debugging on your site and let us know what debug message you get?
    Here’s how you can turn on debugging. https://wordpress.org/support/article/debugging-in-wordpress/

    Regards,
    Biplav

    Thread Starter glashsix

    (@glashsix)

    Hi Biplav,

    I’m sorry, I know this isn’t really helpful without any further details.. Even so, I wanted to let you guys know, that we experienced these issues since I appreciate the work you do.

    There were actually no errors, notices or warnings related to the plugin in our debug.log during the time we knew the plugin was the issue. We also didn’t see related errors in our php logs.

    I suspect it was rather something about hogging up the connections to the site or processes / threads in the server, because it also seemed to affect our load balancing/caching server since our sites behind the same caching server but on different application server were unreachable too.

    It was a really strange issue. If I had any more info I’d pass it on for sure, but I unfortunately can’t reactivate the plugin in production to find out more 🙂

    I appreciate the plugin and hope you manage to find the issue.
    Good luck!

    agathongroup

    (@agathongroup)

    We have a client who experienced this problem this morning. There’s no error or message because the PHP processes just spin before timing out.

    The problem is in the database, which seems to be in a death spiral executing the following query over and over and over:

    OPTIMIZE TABLE wp_blc_links, wp_blc_instances, wp_blc_synch

    Especially on InnoDB, OPTIMIZE TABLE is an extremely expensive operation. Further, it’s blocking other normal queries on the wp_blc_instances table in Waiting for table level lock.

    Unfortunately the only fix appears to be to completely remove the plugin folder, as we can’t even get wp plugin deactivate to work. Incidentally, here’s a partial strace of wp plugin list with timestamps so you can see where it’s hanging:

    1590503980.171024 sendto(3, " \0\0\0\3SHOW INDEX FROMwp_blc_synch`;”, 36, MSG_DONTWAIT, NULL, 0) = 36
    1590503980.171320 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = 1 ([{fd=3, revents=POLLIN}])
    1590503980.172042 recvfrom(3, “\1\0\0\1\rK\0\0\2\3def\22information_schema\nSTATISTICS\nSTATISTICS\5Table\nTABLE_NAME\f\340\0\0\1\0\0\375\1\0\0\0\0P\0\0\3\3def\22information_schema\nSTATISTICS\nSTATISTICS\nNon_unique\nNON_UNIQUE\f?\0\1\0\0\0\10\1\0\0\0\0N\0\0\4\3def\22information_schema\nSTATISTICS\nSTATISTICS\10Key_name\nINDEX_NAME\f\340\0\0\1\0\0\375\1\0\0\0\0T\0\0\5\3def\22information_schema\nSTATISTICS\nSTATISTICS\fSeq_in_index\fSEQ_IN_INDEX\f?\0\2\0\0\0\10\1\0\0\0\0R\0\0\6\3def\22information_schema\nSTATISTICS\nSTATISTICS\vColumn_name\vCOLUMN_NAME\f\340\0\0\1\0\0\375\1\0\0\0\0N\0\0\7\3def\22information_schema\nSTATISTICS\nSTATISTICS\tCollation\tCOLLATION\f\340\0\4\0\0\0\375\0\0\0\0\0R\0\0\10\3def\22information_schema\nSTATISTICS\nSTATISTICS\vCardinality\vCARDINALITY\f?\0\25\0\0\0\10\0\0\0\0\0L\0\0\t\3def\22information_schema\nSTATISTICS\nSTATISTICS\10Sub_part\10SUB_PART\f?\0\3\0\0\0\10\0\0\0\0\0H\0\0\n\3def\22information_schema\nSTATISTICS\nSTATISTICS\6Packed\6PACKED\f\340\0(\0\0\0\375\0\0\0\0\0H\0\0\v\3def\22information_schema\nSTATISTICS\nSTATISTICS\4Null\10NULLABLE\f\340\0\f\0\0\0\375\1\0\0\0\0P\0\0\f\3def\22information_schema\nSTATISTICS\nSTATISTICS\nIndex_type\nINDEX_TYPE\f\340\0@\0\0\0\375\1\0\0\0\0J\0\0\r\3def\22information_schema\nSTATISTICS\nSTATISTICS\7Comment\7COMMENT\f\340\0@\0\0\0\375\0\0\0\0\0V\0\0\16\3def\22information_schema\nSTATISTICS”…, 33666, MSG_DONTWAIT, NULL, NULL) = 1265
    1590503980.172482 gettimeofday({tv_sec=1590503980, tv_usec=172531}, NULL) = 0
    1590503980.172791 gettimeofday({tv_sec=1590503980, tv_usec=172838}, NULL) = 0
    1590503980.173086 sendto(3, “R\0\0\0\3ALTER TABLE wp_blc_instances MODIFY COLUMN link_text text NOT NULL DEFAULT ””, 86, MSG_DONTWAIT, NULL, 0) = 86
    1590503980.173497 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
    1590504245.813742 — SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} —
    1590504246.116273 +++ killed by SIGINT +++`

    Hopefully this helps!

    agathongroup

    (@agathongroup)

    (Sorry, that formatting is hot garbage. Point being this particular command line operation sent ALTER TABLE wp_blc_instances MODIFY COLUMN link_text text NOT NULL DEFAULT '' and then spent five minutes waiting for MySQL to respond b/c it was doing other things.)

    sneader

    (@sneader)

    Just jumping on the bandwagon… we provide hosting services and one of our clients had a site that also went into a death spiral. He said he only updated two plugins before it happened, and yours was one of them. We removed the plugin via command line and it brought the site back up and running. Clearly 1.11.13 has a serious issue. I’m glad that @agathongroup was able to capture so much information to assist you.

    – Scott

    wbmxcs

    (@wbmxcs)

    My site is down. Removed the plugin folder and its still down. How to fix?

    Update: i uploaded older version of the plugin via Cpanel and extracted it after deleting the old folder and my site is back online.

    • This reply was modified 4 years ago by wbmxcs.
    blogodisea

    (@blogodisea)

    My site also was offline when i updated the plugin, it got stucked. I also deactivated it till there’s a solution. Good plugin and good work, a pity the plugin had a problem today.

    Hello everyone, ( @sneader @blogodisea @wbmxcs @agathongroup @glashsix )

    We had made a certain fix to the way the plugin is handling database in this update to make a foundation for the huge new shiny update coming up which improves the UI and the over all performance of the plugin.

    For now the plugin might not be handling the update gracefully because there are huge amount of links on your database or because some of the queries are expensive which on limited resources can cause timeouts.

    The reason for this is WPMU DEV just acquired the plugin and have been trying to enhance the plugin’s performance, particularly the DB update code is quite old and we’re trying our best to safely update it.

    We will try to take care of this DB upgrade on the future updates but if you want a hot fix right now, you’ll have to uninstall the plugin from WP Admin panel ( which will delete the plugin’s custom tables & options ) then installing the new version should do the trick.

    I will have to warn you that this will delete plugin options and will recheck the all the links again once the plugin is installed so it would be safe to keep backup of your DB if you want to rollback.

    Unfortunately, I can’t recreate this issue in any of my test servers, so if someone could provide me with the server configuration details and if possible the number of links on the site I could try and recreate it and maybe release a quick fix too.

    Sorry for the inconvenience and thank you for your feedbacks and reports.
    Best regards,
    Biplav.

    blogodisea

    (@blogodisea)

    Hello, thanks for fixing it soon. The problem is that i can’t deactivate the plugin as the site was broken. I assume that this will happen to more people, don’t know if we can access the plugin panel while the plugin is working, so it will be impossible to deactivate.

    I had to take off the Broken Link Checker folder from the plugin folder to activate the site again.

    So the tables blc_filters, blc_instances, blc_links and blc_synch (and there will be in option table too i imagine) are still there.

    Can we just delete these 4 tables and install the plugin again? Don’t know if the options table that keep old records of the plugin will make any problem.

    Thanks a lot for your work.

    blogodisea

    (@blogodisea)

    I deleted the blc_filters, blc_instances, blc_links and blc_synch tables from the database and reinstalled the plugin again.

    I checked the options and selected the “nuclear option” for resetting the plugin so it deletes the link database and re-check the whole site from scratch.

    In this moment is working fine, don’t know if later will paralyze.

    archon810

    (@archon810)

    Looks like the same issue we warned you about here after the last update https://wordpress.org/support/topic/db-migration-run-to-often/.

    @blogodisea Glad that’s working for you. I don’t think you’ll face those issues now since the table is already updated. But if you do please let us know.

    @archon810 Yeah that’s where the whole process started, we’re trying to fix that but that’s a long process and with trying to do it safely for all our users.
    I’ve replied on the tread with more details and how we’re handling these more on the future updates.

    We’re really sorry for any inconvenience caused,
    Best regards.
    Biplav.

    Plugin Support Pawel – WPMU DEV Support

    (@wpmudev-support9)

    Hello @glashsix

    I hope you are doing well!

    We haven’t heard back from you for a while now so we’ve marked this ticket as resolved. If you do have any followup questions or require further assistance feel free to reopen it and let us know here.

    Kind regards,
    Nastia

    Thread Starter glashsix

    (@glashsix)

    Hi Nastia @wpmudev-support9,

    how are you? Hope you are doing well too! 🙂

    Sorry for not getting back to you on the issue, I felt like others provided more valuable feedback and then didn’t get the chance…

    Ofcourse, I hope this helped and if it doesn’t need to stay open for people to send their case feel free to close the topic.

    Thank You
    Bests

    I momentarily forgot about this and updated the plugin on our site, and all hell broke loose, taking down our site and making the db slaves lag for an hour after I disabled the plugin due to slaves executing the hundreds of OPTIMIZE queries one by one.

    View post on imgur.com

    View post on imgur.com

    View post on imgur.com

    View post on imgur.com

    What’s been the progress of preventing updates from trashing the database?

Viewing 15 replies - 1 through 15 (of 27 total)
  • The topic ‘Update 1.11.13 broke the site’ is closed to new replies.