Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.................XX..XX..X..X...X....XX..X...X.X.XXXXXXXXXXX] duration: 30.001395508s trace: https://jaeger.dev.storj.io/trace/1e55be1a2bac79ae FAIL (2025-12-17 14:06:19 UTC) orbiter warning: context deadline exceeded Duration: 30.001421159s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X...................X..........X...X...............X..] duration: 24.10474145s trace: https://jaeger.dev.storj.io/trace/2e9824ebdc9568e6 FAIL (2025-12-17 14:06:12 UTC) orbiter warning: context deadline exceeded Duration: 24.10475642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X...................X..........X...X...............X.] duration: 39.346742511s trace: https://jaeger.dev.storj.io/trace/2e9824ebdc9568e6 FAIL (2025-12-17 14:05:48 UTC) orbiter warning: context deadline exceeded Duration: 39.346758091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................X........................................] duration: 6.040135846s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:05:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.040164439s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..................XX..XX..X..X...X....XX..X...X.X.XXXXXXXXXX] duration: 30.000786966s trace: https://jaeger.dev.storj.io/trace/fee84e427285f6c FAIL (2025-12-17 14:05:27 UTC) orbiter warning: context deadline exceeded Duration: 30.000813837s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXXXXXXX] duration: 5.001213904s trace: https://jaeger.dev.storj.io/trace/152f8306fec363ee FAIL (2025-12-17 14:05:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001243531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X...................X..........X...X...............X] duration: 1m0.001756843s trace: https://jaeger.dev.storj.io/trace/2e9824ebdc9568e6 FAIL (2025-12-17 14:05:08 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001779377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...................XX..XX..X..X...X....XX..X...X.X.XXXXXXXXX] duration: 30.000877784s trace: https://jaeger.dev.storj.io/trace/18f6490efd39783 FAIL (2025-12-17 14:04:43 UTC) orbiter warning: context deadline exceeded Duration: 30.00089129s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................X.......................................] duration: 6.07650488s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:04:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.076530449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................X......................................] duration: 5.79569342s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:04:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.795714741s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....................XX..XX..X..X...X....XX..X...X.X.XXXXXXXX] duration: 30.001750248s trace: https://jaeger.dev.storj.io/trace/451f4a35eacc67fd FAIL (2025-12-17 14:04:13 UTC) orbiter warning: context deadline exceeded Duration: 30.001800904s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXXXXXX] duration: 5.002284585s trace: https://jaeger.dev.storj.io/trace/a2040263d771b86 FAIL (2025-12-17 14:04:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002310053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXXXXX] duration: 5.001144172s trace: https://jaeger.dev.storj.io/trace/15bc25a63110e315 FAIL (2025-12-17 14:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001173277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................X.....................................] duration: 6.161548277s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.161577643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....................XX..XX..X..X...X....XX..X...X.X.XXXXXXX] duration: 30.001252501s trace: https://jaeger.dev.storj.io/trace/6bec75dd1bde1dc7 FAIL (2025-12-17 14:03:09 UTC) orbiter warning: context deadline exceeded Duration: 30.001270976s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...................X..........X...X...............] duration: 8.704802502s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 14:03:04 UTC) orbiter warning: context deadline exceeded Duration: 8.704828832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXXXX] duration: 5.000527337s trace: https://jaeger.dev.storj.io/trace/433235f6efd18c29 FAIL (2025-12-17 14:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000553268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X................X....................................] duration: 6.231018571s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.231047928s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......................XX..XX..X..X...X....XX..X...X.X.XXXXXX] duration: 30.00133201s trace: https://jaeger.dev.storj.io/trace/5518d10e650b7213 FAIL (2025-12-17 14:02:31 UTC) orbiter warning: context deadline exceeded Duration: 30.001407922s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...................X..........X...X..............] duration: 32.029762065s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 14:02:29 UTC) orbiter warning: context deadline exceeded Duration: 32.029888637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X................X...................................] duration: 13.380407615s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:02:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 13.380438342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXXX] duration: 5.001639498s trace: https://jaeger.dev.storj.io/trace/50e6a7b3727bd287 FAIL (2025-12-17 14:02:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001694232s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......................XX..XX..X..X...X....XX..X...X.X.XXXXX] duration: 30.001461495s trace: https://jaeger.dev.storj.io/trace/4a1edd4679241978 FAIL (2025-12-17 14:01:55 UTC) orbiter warning: context deadline exceeded Duration: 30.001483287s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................X..................................] duration: 6.168416534s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.168447243s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X...................X..........X...X.............] duration: 6.696186437s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 14:01:15 UTC) orbiter warning: context deadline exceeded Duration: 6.696212678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................X.................................] duration: 5.954723434s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 14:00:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.954749503s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXXX] duration: 5.001502404s trace: https://jaeger.dev.storj.io/trace/25b08f2b7049a210 FAIL (2025-12-17 14:00:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001527651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........................XX..XX..X..X...X....XX..X...X.X.XXXX] duration: 10.600548203s trace: https://jaeger.dev.storj.io/trace/4e934b7f9e42cf6c FAIL (2025-12-17 14:00:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.600566958s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...................X..........X...X............] duration: 37.248306537s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 14:00:42 UTC) orbiter warning: context deadline exceeded Duration: 37.248344238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXXX] duration: 5.001480414s trace: https://jaeger.dev.storj.io/trace/7308a19b9fddf73e FAIL (2025-12-17 14:00:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001512994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...................X..........X...X...........] duration: 26.580217496s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 14:00:05 UTC) orbiter warning: context deadline exceeded Duration: 26.580236993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........................XX..XX..X..X...X....XX..X...X.X.XXX] duration: 30.001405089s trace: https://jaeger.dev.storj.io/trace/184d8330e61e74dd FAIL (2025-12-17 14:00:01 UTC) orbiter warning: context deadline exceeded Duration: 30.001419206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................X................................] duration: 5.817620737s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:59:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.817719707s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXXX] duration: 5.001259805s trace: https://jaeger.dev.storj.io/trace/854693e60854c77 FAIL (2025-12-17 13:59:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00128331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...................X..........X...X..........] duration: 24.54490559s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 13:59:38 UTC) orbiter warning: context deadline exceeded Duration: 24.544920259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................XX..XX..X..X...X....XX..X...X.X.XX] duration: 30.001183549s trace: https://jaeger.dev.storj.io/trace/7d76cc9042c39413 FAIL (2025-12-17 13:59:31 UTC) orbiter warning: context deadline exceeded Duration: 30.00120523s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X................X...............................] duration: 6.097511547s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.097545201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...................X..........X...X.........] duration: 32.050837237s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 13:59:13 UTC) orbiter warning: context deadline exceeded Duration: 32.050851976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X................X..............................] duration: 6.053415827s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.053445343s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...................X..........X...X........] duration: 41.332929366s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 13:58:41 UTC) orbiter warning: context deadline exceeded Duration: 41.332960786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XXX] duration: 5.001325685s trace: https://jaeger.dev.storj.io/trace/71063e43cf7e2b9d FAIL (2025-12-17 13:58:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001341295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X................X.............................] duration: 6.054416592s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.054445387s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XX..XX..X..X...X....XX..X...X.X.X] duration: 30.00134293s trace: https://jaeger.dev.storj.io/trace/703c3d03ba85029d FAIL (2025-12-17 13:58:18 UTC) orbiter warning: context deadline exceeded Duration: 30.001364901s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X................X............................] duration: 6.186633618s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:58:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.18666063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.XXXXXXXX.XX.X..XXXXXXX.XX] duration: 5.001437469s trace: https://jaeger.dev.storj.io/trace/7e7df55b4f458860 FAIL (2025-12-17 13:58:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001468056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X................X...........................] duration: 5.952315001s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:57:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.952330842s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X...................X..........X...X.......] duration: 27.207615362s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 13:57:22 UTC) orbiter warning: context deadline exceeded Duration: 27.207631914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............................XX..XX..X..X...X....XX..X...X.X.] duration: 26.171441073s trace: https://jaeger.dev.storj.io/trace/d878d2f5021f346 FAIL (2025-12-17 13:57:20 UTC) orbiter warning: context deadline exceeded Duration: 26.171486699s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X................X..........................] duration: 5.948315209s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:57:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.948345285s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X...................X..........X...X......] duration: 33.003421938s trace: https://jaeger.dev.storj.io/trace/63f571d7271ddaf5 FAIL (2025-12-17 13:56:55 UTC) orbiter warning: context deadline exceeded Duration: 33.00345452s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)