Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.856826358s
FAIL (2025-07-13 16:47:41 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.856852959s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.867676978s
FAIL (2025-07-13 16:47:05 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.867696515s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.205771831s
FAIL (2025-07-13 16:46:09 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.205799704s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.915429937s
FAIL (2025-07-13 16:45:18 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.915488327s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
1.913861436s
FAIL (2025-07-13 16:44:42 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.913887708s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.844712292s
FAIL (2025-07-13 16:43:45 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.844736208s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
1.935329527s
FAIL (2025-07-13 16:42:40 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.935353103s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
1.877465174s
FAIL (2025-07-13 16:41:45 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.877492697s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
1.878536313s
FAIL (2025-07-13 16:40:40 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.878562262s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.151296882s
FAIL (2025-07-13 16:39:29 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.151318582s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
1.979968069s
FAIL (2025-07-13 16:38:19 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.979990142s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
1.894984571s
FAIL (2025-07-13 16:37:10 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.895014249s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
1.862974722s
FAIL (2025-07-13 16:36:50 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.863001943s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
1.852639092s
FAIL (2025-07-13 16:35:43 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.85267974s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
1.835833349s
FAIL (2025-07-13 16:34:50 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.835868457s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
1.83564037s
FAIL (2025-07-13 16:33:37 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.83567153s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
1.884894321s
FAIL (2025-07-13 16:32:27 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.884923617s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
1.880940485s
FAIL (2025-07-13 16:31:37 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.880969953s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
1.849719287s
FAIL (2025-07-13 16:30:29 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.849747379s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
1.845152389s
FAIL (2025-07-13 16:29:55 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.845182616s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
1.88053896s
FAIL (2025-07-13 16:28:55 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.88056558s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
1.869515038s
FAIL (2025-07-13 16:27:52 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.869556308s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
1.891586966s
FAIL (2025-07-13 16:27:21 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.891613317s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.048787418s
FAIL (2025-07-13 16:26:13 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.048817084s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
1.869861541s
FAIL (2025-07-13 16:25:02 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.869888965s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.287269064s
FAIL (2025-07-13 16:24:44 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.287301555s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
1.943217635s
FAIL (2025-07-13 16:24:08 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.94324602s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
1.862077218s
FAIL (2025-07-13 16:23:38 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.862102205s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
1.942173647s
FAIL (2025-07-13 16:23:02 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.942209797s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
1.840539444s
FAIL (2025-07-13 16:22:17 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.840566695s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
1.896888559s
FAIL (2025-07-13 16:21:56 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.89691544s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
1.875069596s
FAIL (2025-07-13 16:21:18 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.875099905s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
1.929354072s
FAIL (2025-07-13 16:20:38 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.92937892s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
1.898227322s
FAIL (2025-07-13 16:20:06 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.898286884s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
1.848057938s
FAIL (2025-07-13 16:19:27 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.848087745s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
1.890060264s
FAIL (2025-07-13 16:18:56 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.890082358s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
1.937094693s
FAIL (2025-07-13 16:17:53 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.937132676s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.917390114s
FAIL (2025-07-13 16:16:45 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.917421985s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.056682271s
FAIL (2025-07-13 16:15:59 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.056709644s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.074090467s
FAIL (2025-07-13 16:15:14 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.07411832s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.858385411s
FAIL (2025-07-13 16:14:31 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.85840591s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.8569953s
FAIL (2025-07-13 16:13:56 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.85702068s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.119324695s
FAIL (2025-07-13 16:13:04 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.11935246s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.070366725s
FAIL (2025-07-13 16:11:54 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 2.070400669s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.86369057s
FAIL (2025-07-13 16:10:47 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.863716299s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.706041675s
FAIL (2025-07-13 16:10:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.70607085s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.827371765s
FAIL (2025-07-13 16:10:00 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.827392505s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.793975058s
FAIL (2025-07-13 16:09:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.793993201s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.877167052s
FAIL (2025-07-13 16:09:01 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== load Jul 13 15:54:46.108871399 (486h44m20.879512734s) [upload=389315]: Done starting upload Jul 13 15:54:46.108943557 (486h44m20.879584892s) [upload=389315]: Uploading segments... Jul 13 15:54:46.108977262 (486h44m20.879618597s) [upload=389315]: Got next segment. Inline: true Jul 13 15:54:46.108981219 (486h44m20.879622554s) [upload=389315]: Waiting for error group managing segments... Jul 13 15:54:46.109017639 (486h44m20.879658974s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:54:46.109022589 (486h44m20.879663924s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:54:46.109025304 (486h44m20.879666639s) [upload=389315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:54:46.386239686 (486h44m21.156881031s) [upload=389315]: Done uploading segments... Jul 13 15:54:46.386269193 (486h44m21.156910528s) [upload=389315]: Upload finished. err: <nil> Jul 13 15:55:40.344470993 (486h45m15.115112328s) [upload=389328]: Starting upload Jul 13 15:55:40.344550375 (486h45m15.11519171s) [upload=389328]: Done starting upload Jul 13 15:55:40.344652100 (486h45m15.115293465s) [upload=389328]: Uploading segments... Jul 13 15:55:40.344712545 (486h45m15.11535392s) [upload=389328]: Got next segment. Inline: true Jul 13 15:55:40.344720150 (486h45m15.115361515s) [upload=389328]: Waiting for error group managing segments... Jul 13 15:55:40.344778762 (486h45m15.115420107s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:55:40.344787619 (486h45m15.115428954s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:55:40.344790805 (486h45m15.11543214s) [upload=389328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:55:40.623795121 (486h45m15.394436456s) [upload=389328]: Done uploading segments... Jul 13 15:55:40.623836190 (486h45m15.394477525s) [upload=389328]: Upload finished. err: <nil> Jul 13 15:56:54.182246759 (486h46m28.952888094s) [upload=389343]: Starting upload Jul 13 15:56:54.182305641 (486h46m28.952946977s) [upload=389343]: Done starting upload Jul 13 15:56:54.182351328 (486h46m28.952992663s) [upload=389343]: Uploading segments... Jul 13 15:56:54.182381687 (486h46m28.953023012s) [upload=389343]: Got next segment. Inline: true Jul 13 15:56:54.182385254 (486h46m28.953026579s) [upload=389343]: Waiting for error group managing segments... Jul 13 15:56:54.182415882 (486h46m28.953057217s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:56:54.182420331 (486h46m28.953061666s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:56:54.182422585 (486h46m28.95306392s) [upload=389343]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:56:54.468829364 (486h46m29.239470699s) [upload=389343]: Done uploading segments... Jul 13 15:56:54.468847930 (486h46m29.239489265s) [upload=389343]: Upload finished. err: <nil> Jul 13 15:58:03.662420515 (486h47m38.43306185s) [upload=389361]: Starting upload Jul 13 15:58:03.662481121 (486h47m38.433122456s) [upload=389361]: Done starting upload Jul 13 15:58:03.662551736 (486h47m38.433193071s) [upload=389361]: Uploading segments... Jul 13 15:58:03.662647138 (486h47m38.433288473s) [upload=389361]: Got next segment. Inline: true Jul 13 15:58:03.662652058 (486h47m38.433293393s) [upload=389361]: Waiting for error group managing segments... Jul 13 15:58:03.662713356 (486h47m38.433354691s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:03.662718996 (486h47m38.433360331s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:03.662722012 (486h47m38.433363347s) [upload=389361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:04.014783378 (486h47m38.785424713s) [upload=389361]: Done uploading segments... Jul 13 15:58:04.014799388 (486h47m38.785440723s) [upload=389361]: Upload finished. err: <nil> Jul 13 15:58:31.000045680 (486h48m5.770687015s) [upload=389368]: Starting upload Jul 13 15:58:31.000113600 (486h48m5.770754925s) [upload=389368]: Done starting upload Jul 13 15:58:31.000272984 (486h48m5.770914329s) [upload=389368]: Uploading segments... Jul 13 15:58:31.000434754 (486h48m5.771076109s) [upload=389368]: Got next segment. Inline: true Jul 13 15:58:31.000442729 (486h48m5.771084064s) [upload=389368]: Waiting for error group managing segments... Jul 13 15:58:31.000532702 (486h48m5.771174047s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:31.000542761 (486h48m5.771184096s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:31.000546748 (486h48m5.771188083s) [upload=389368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:31.279315035 (486h48m6.04995641s) [upload=389368]: Done uploading segments... Jul 13 15:58:31.279338460 (486h48m6.049979815s) [upload=389368]: Upload finished. err: <nil> Jul 13 15:58:57.803093746 (486h48m32.573735071s) [upload=389373]: Starting upload Jul 13 15:58:57.803144844 (486h48m32.573786179s) [upload=389373]: Done starting upload Jul 13 15:58:57.803198046 (486h48m32.573839381s) [upload=389373]: Uploading segments... Jul 13 15:58:57.803227161 (486h48m32.573868496s) [upload=389373]: Got next segment. Inline: true Jul 13 15:58:57.803230578 (486h48m32.573871913s) [upload=389373]: Waiting for error group managing segments... Jul 13 15:58:57.803259313 (486h48m32.573900648s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:58:57.803263490 (486h48m32.573904825s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:58:57.803265885 (486h48m32.57390722s) [upload=389373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:58:58.308341968 (486h48m33.078983293s) [upload=389373]: Done uploading segments... Jul 13 15:58:58.308359793 (486h48m33.079001118s) [upload=389373]: Upload finished. err: <nil> Jul 13 15:59:43.717160756 (486h49m18.487802081s) [upload=389385]: Starting upload Jul 13 15:59:43.717224758 (486h49m18.487866094s) [upload=389385]: Done starting upload Jul 13 15:59:43.717275816 (486h49m18.487917141s) [upload=389385]: Uploading segments... Jul 13 15:59:43.717306805 (486h49m18.48794813s) [upload=389385]: Got next segment. Inline: true Jul 13 15:59:43.717310563 (486h49m18.487951888s) [upload=389385]: Waiting for error group managing segments... Jul 13 15:59:43.717342414 (486h49m18.487983749s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:43.717347584 (486h49m18.487988909s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:43.717351421 (486h49m18.487992756s) [upload=389385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:44.030128334 (486h49m18.800769669s) [upload=389385]: Done uploading segments... Jul 13 15:59:44.030150106 (486h49m18.800791441s) [upload=389385]: Upload finished. err: <nil> Jul 13 15:59:58.743009541 (486h49m33.513650876s) [upload=389390]: Starting upload Jul 13 15:59:58.743057132 (486h49m33.513698467s) [upload=389390]: Done starting upload Jul 13 15:59:58.743099995 (486h49m33.51374133s) [upload=389390]: Uploading segments... Jul 13 15:59:58.743128329 (486h49m33.513769654s) [upload=389390]: Got next segment. Inline: true Jul 13 15:59:58.743131555 (486h49m33.51377288s) [upload=389390]: Waiting for error group managing segments... Jul 13 15:59:58.743157975 (486h49m33.51379931s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 15:59:58.743162384 (486h49m33.513803719s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 15:59:58.743164628 (486h49m33.513805963s) [upload=389390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 15:59:59.052447722 (486h49m33.823089057s) [upload=389390]: Done uploading segments... Jul 13 15:59:59.052459595 (486h49m33.82310092s) [upload=389390]: Upload finished. err: <nil> Jul 13 16:00:47.777149861 (486h50m22.547791186s) [upload=389402]: Starting upload Jul 13 16:00:47.777208903 (486h50m22.547850238s) [upload=389402]: Done starting upload Jul 13 16:00:47.777265191 (486h50m22.547906516s) [upload=389402]: Uploading segments... Jul 13 16:00:47.777297553 (486h50m22.547938888s) [upload=389402]: Got next segment. Inline: true Jul 13 16:00:47.777301380 (486h50m22.547942715s) [upload=389402]: Waiting for error group managing segments... Jul 13 16:00:47.777333241 (486h50m22.547974576s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:00:47.777339402 (486h50m22.547980737s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:00:47.777341748 (486h50m22.547983083s) [upload=389402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:00:48.315202595 (486h50m23.08584393s) [upload=389402]: Done uploading segments... Jul 13 16:00:48.315221921 (486h50m23.085863246s) [upload=389402]: Upload finished. err: <nil> Jul 13 16:01:41.500642721 (486h51m16.271284056s) [upload=389416]: Starting upload Jul 13 16:01:41.500744706 (486h51m16.271386041s) [upload=389416]: Done starting upload Jul 13 16:01:41.500806174 (486h51m16.271447509s) [upload=389416]: Uploading segments... Jul 13 16:01:41.500837604 (486h51m16.271478939s) [upload=389416]: Got next segment. Inline: true Jul 13 16:01:41.500841311 (486h51m16.271482646s) [upload=389416]: Waiting for error group managing segments... Jul 13 16:01:41.500874174 (486h51m16.271515509s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 13 16:01:41.500878933 (486h51m16.271520258s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 13 16:01:41.500881377 (486h51m16.271522712s) [upload=389416]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 13 16:01:41.784662533 (486h51m16.555303868s) [upload=389416]: Done uploading segments... Jul 13 16:01:41.784680327 (486h51m16.555321652s) [upload=389416]: Upload finished. err: <nil>
Duration: 1.877200285s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.266164683s
FAIL (2025-07-13 16:08:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.266187808s
edge
gateway
us-select-1
|