Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 1m0.001061094s trace: https://jaeger.dev.storj.io/trace/7182e3a78f502dcb FAIL (2025-12-17 13:42:27 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001083506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [..............XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.X] duration: 5.001263397s trace: https://jaeger.dev.storj.io/trace/61437865709045d1 FAIL (2025-12-17 13:42:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001289686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X................X.........] duration: 6.100820749s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:42:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.100850185s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX..XX..X..X] duration: 30.000541051s trace: https://jaeger.dev.storj.io/trace/19325b373ce903a1 FAIL (2025-12-17 13:41:40 UTC) orbiter warning: context deadline exceeded Duration: 30.000553746s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...............XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X.] duration: 4.031692091s trace: https://jaeger.dev.storj.io/trace/295c0a7a919e1a1d FAIL (2025-12-17 13:41:35 UTC) orbiter warning: context deadline exceeded Duration: 4.03171759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X................X........] duration: 6.036594214s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:41:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.03662414s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX..XX..X..] duration: 28.931071825s trace: https://jaeger.dev.storj.io/trace/79c62ed65fe96342 FAIL (2025-12-17 13:41:10 UTC) orbiter warning: context deadline exceeded Duration: 28.931106872s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 34.148880719s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:41:02 UTC) orbiter warning: context deadline exceeded Duration: 34.148914602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.X] duration: 5.000871633s trace: https://jaeger.dev.storj.io/trace/295c0a7a919e1a1d FAIL (2025-12-17 13:40:58 UTC) orbiter warning: context deadline exceeded Duration: 5.000899886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X................X.......] duration: 5.952726686s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:40:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.952754139s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XX..XX..X.] duration: 27.41047791s trace: https://jaeger.dev.storj.io/trace/79c62ed65fe96342 FAIL (2025-12-17 13:40:38 UTC) orbiter warning: context deadline exceeded Duration: 27.410507248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X................X......] duration: 6.10131151s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:40:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.101343761s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 26.001036219s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:40:26 UTC) orbiter warning: context deadline exceeded Duration: 26.001065514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 24.32184671s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:39:54 UTC) orbiter warning: context deadline exceeded Duration: 24.321863291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX.] duration: 4.771907102s trace: https://jaeger.dev.storj.io/trace/608904ef162d0145 FAIL (2025-12-17 13:39:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.771938332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX..XX..X] duration: 30.00158123s trace: https://jaeger.dev.storj.io/trace/79c62ed65fe96342 FAIL (2025-12-17 13:39:35 UTC) orbiter warning: context deadline exceeded Duration: 30.001605415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration: 6.098980231s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.099018345s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 24.928623599s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:39:29 UTC) orbiter warning: context deadline exceeded Duration: 24.928652085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXXX] duration: 5.000759391s trace: https://jaeger.dev.storj.io/trace/608904ef162d0145 FAIL (2025-12-17 13:39:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001102385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 10.085698185s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:38:39 UTC) orbiter warning: context deadline exceeded Duration: 10.08573322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X................X....] duration: 5.91131604s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:38:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.911353631s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX..XX..] duration: 24.853650585s trace: https://jaeger.dev.storj.io/trace/4af6b4e6d13c0deb FAIL (2025-12-17 13:38:21 UTC) orbiter warning: context deadline exceeded Duration: 24.853856468s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [...................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XXX] duration: 5.001487895s trace: https://jaeger.dev.storj.io/trace/353288dbdf3d876a FAIL (2025-12-17 13:38:13 UTC) orbiter warning: context deadline exceeded Duration: 5.00151188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 18.083209617s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:38:00 UTC) orbiter warning: context deadline exceeded Duration: 18.083232852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X................X...] duration: 6.053534443s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 6.05355935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X................X..] duration: 5.983931166s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.983955403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.XX] duration: 5.000894468s trace: https://jaeger.dev.storj.io/trace/55a68597af9ba28 FAIL (2025-12-17 13:37:13 UTC) orbiter warning: context deadline exceeded Duration: 5.000935685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 41.189732273s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:37:10 UTC) orbiter warning: context deadline exceeded Duration: 41.189762661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....................................................XX..XX.] duration: 21.320966698s trace: https://jaeger.dev.storj.io/trace/4af6b4e6d13c0deb FAIL (2025-12-17 13:37:09 UTC) orbiter warning: context deadline exceeded Duration: 21.320983278s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....................................................XX..XX] duration: 30.001103284s trace: https://jaeger.dev.storj.io/trace/4af6b4e6d13c0deb FAIL (2025-12-17 13:36:48 UTC) orbiter warning: context deadline exceeded Duration: 30.001126689s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X................X.] duration: 5.768351964s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 5.76838199s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....................................................XX..X] duration: 30.001680989s trace: https://jaeger.dev.storj.io/trace/67487f635732896d FAIL (2025-12-17 13:36:06 UTC) orbiter warning: context deadline exceeded Duration: 30.001708562s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.X] duration: 5.002459676s trace: https://jaeger.dev.storj.io/trace/28ae293e5fa7b2c6 FAIL (2025-12-17 13:36:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002489672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X................X] duration: 30.002208603s trace: https://jaeger.dev.storj.io/trace/3785adf9c853abf8 FAIL (2025-12-17 13:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA) context deadline exceeded ======LOGS====== Dec 17 13:35:19.492165075 (286h6m19.36324811s) [upload=250856]: Starting upload Dec 17 13:35:19.492221884 (286h6m19.363304929s) [upload=250856]: Done starting upload Dec 17 13:35:19.492329609 (286h6m19.363412644s) [upload=250856]: Uploading segments... Dec 17 13:35:19.492417687 (286h6m19.363500722s) [upload=250856]: Got next segment. Inline: false Dec 17 13:35:19.492452613 (286h6m19.363535648s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:35:19.492458655 (286h6m19.3635417s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:35:19.745381105 (286h6m19.61646416s) [seg_pos={0 0}, upload=250856]: Begin upload segment... Dec 17 13:35:19.745442542 (286h6m19.616525577s) [seg_pos={0 0}, upload=250856]: Done begin upload segment. Dec 17 13:35:19.745501284 (286h6m19.616584329s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.745523416 (286h6m19.616606461s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746145741 (286h6m19.617228786s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.746605086 (286h6m19.617688131s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747184469 (286h6m19.618267514s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.747710361 (286h6m19.618793406s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.748230691 (286h6m19.619313736s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749393996 (286h6m19.620477031s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.749909468 (286h6m19.620992523s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.750570497 (286h6m19.621653532s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751588024 (286h6m19.622671069s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.751646616 (286h6m19.622729651s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.753246723 (286h6m19.624329768s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.754901022 (286h6m19.625984057s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.756569720 (286h6m19.627652775s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:19.759490812 (286h6m19.630573857s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Uploading piece... Dec 17 13:35:21.209116402 (286h6m21.080199447s) [noise=true , address=77.90.31.21:4877 , piece_id=SJVPA5NQ4U6CJ4MPGGNSNBP3MMFQOHX4GWH7TWRFIFENDTUGB5LA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 13:35:21.469050487 (286h6m21.340133532s) [upload=250856]: Next returned nil segment Dec 17 13:35:21.469074192 (286h6m21.340157227s) [upload=250856]: Waiting for error group managing segments... Dec 17 13:35:21.746241891 (286h6m21.617324926s) [noise=true , address=145.79.227.49:4339 , piece_id=BHW5GCJI65LXDDG3AGBY6FYLN6G2K2H63SWST32BEDDGZAB5JSIA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.746387538 (286h6m21.617470573s) [noise=true , address=145.79.227.40:4051 , piece_id=NOB3AZW5PLOFWLNQEH2WPAACUGVG7JA262AHCOQ4SP2TKZDAK7ZA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747197241 (286h6m21.618280276s) [noise=true , address=145.79.227.34:4906 , piece_id=QAEVTNFKSZ22D6NJ75N5YNJXBW6ZAWHJV6RYIM7PJSXZHHTZN6TA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.747331046 (286h6m21.618414101s) [noise=true , address=145.79.227.36:4519 , piece_id=4U76MTAKFK4IP4F7KM3D3AEVNRPWMQLGTG7OASQH76EVTTXCULFA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.765698961 (286h6m21.636781996s) [noise=true , address=145.79.227.24:4581 , piece_id=B7C2TYIIY2NRZIL7Y45ZJ4DMZDDG67GBT47ZCJZETRAXDGBNZJAQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.766329532 (286h6m21.637412578s) [noise=true , address=145.79.227.52:4446 , piece_id=SNAIIYCMDLB7725ZD2EIAAISJ6ZOAU2S7NCT7YB6KMODU3M7U3VQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.844401585 (286h6m21.71548462s) [noise=true , address=145.79.227.11:4732 , piece_id=DDQODZRSRBVR72S33ZDDXK55JPEA3NRAVRJHGA3CS4M5766RIC7Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.850001549 (286h6m21.721084594s) [noise=true , address=145.79.227.25:4617 , piece_id=M2QA7PPBZKGZQL5KIUZ5JJVMLBVTD5Q4XM57KGFH5O2YFKSGSCDA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.884325859 (286h6m21.755408904s) [noise=true , address=145.79.227.39:4014 , piece_id=EIMXMBU4KR534DBMP76JUUQQ3SMDKFLLJYAUFOCQI6YBWFYTHYNQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:21.889191785 (286h6m21.76027482s) [noise=true , address=145.79.227.23:4547 , piece_id=XN2G5QBPXC6TF54U5OODVD7TYWIU7P5HOS7NEBLMTRF2IWQ6LD2Q, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.007491640 (286h6m21.878574685s) [noise=true , address=45.91.116.6:4975 , piece_id=6T5KVMFV6Q7K5LIU2GHLMKZMMIP6EW4UVGBPA2ZK6WVSXINMOBGQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.039320929 (286h6m21.910403974s) [noise=true , address=45.91.116.14:4189 , piece_id=2EMQG2W2QOPDAOMMHYVK5QGQ4FYLV25WRJS3H3EVSAFGQAGJAIPQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042055897 (286h6m21.913138942s) [noise=true , address=77.90.31.13:4629 , piece_id=LW2A2DIMFRDHVVDM2UVV2JCLPMBHW356SSVQJRDPWXNQCPT2AAVA, seg_pos={0 0}, upload=250856]: Done uploading piece. err: <nil> Dec 17 13:35:22.042077147 (286h6m21.913160182s) [seg_pos={0 0}, upload=250856]: Segment reached optimal threshold of 13 pieces. Dec 17 13:35:22.042272429 (286h6m21.913355474s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042280695 (286h6m21.91336373s) [noise=true , address=45.91.116.12:4226 , piece_id=EGHEBNXBJOCUAV46AZ24L6FCCSCGLWZEZGZJVUNUCBNS45WGKLHQ, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042383952 (286h6m21.913466987s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 17 13:35:22.042393039 (286h6m21.913476074s) [noise=true , address=45.91.116.5:4300 , piece_id=MKDKBTNRTH2GYOIE7UNGVK6TGMXLKG7B7FUIQEWUGJNRA4BHXQ4A, seg_pos={0 0}, upload=250856]: Long tail cancellation due to optimal threshold Dec 17 13:35:22.042414099 (286h6m21.913497134s) [seg_pos={0 0}, upload=250856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:35:22.042514090 (286h6m21.913597125s) [upload=250856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:35:22.042519710 (286h6m21.913602745s) [upload=250856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:35:22.333256488 (286h6m22.204339523s) [upload=250856]: Done uploading segments... Dec 17 13:35:22.333284061 (286h6m22.204367096s) [upload=250856]: Upload finished. err: <nil> Duration: 30.00223364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 27.6419848s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:35:47 UTC) orbiter warning: context deadline exceeded Duration: 27.642025619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX.] duration: 3.643111421s trace: https://jaeger.dev.storj.io/trace/738cf1b656998eaa FAIL (2025-12-17 13:35:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.643131961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....................................................XX..] duration: 21.717742825s trace: https://jaeger.dev.storj.io/trace/93237fddcef4f66 FAIL (2025-12-17 13:34:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.717773222s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.XX] duration: 5.002001206s trace: https://jaeger.dev.storj.io/trace/738cf1b656998eaa FAIL (2025-12-17 13:34:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002023779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.047721473s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:34:33 UTC) orbiter warning: context deadline exceeded Duration: 7.047745849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.948809537s trace: https://jaeger.dev.storj.io/trace/6aa3dce83f9968cb FAIL (2025-12-17 13:34:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 13:23:20.683120844 (285h54m20.554203889s) [upload=250677]: Starting upload Dec 17 13:23:20.683181839 (285h54m20.554264874s) [upload=250677]: Done starting upload Dec 17 13:23:20.683290607 (285h54m20.554373652s) [upload=250677]: Uploading segments... Dec 17 13:23:20.683694366 (285h54m20.554777411s) [upload=250677]: Got next segment. Inline: false Dec 17 13:23:20.683749731 (285h54m20.554832767s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:23:20.683755452 (285h54m20.554838487s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:23:20.999940787 (285h54m20.871023832s) [seg_pos={0 0}, upload=250677]: Begin upload segment... Dec 17 13:23:21.000001873 (285h54m20.871084938s) [seg_pos={0 0}, upload=250677]: Done begin upload segment. Dec 17 13:23:21.000039866 (285h54m20.871122901s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001756816 (285h54m20.872839861s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001853980 (285h54m20.872937025s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002271615 (285h54m20.87335465s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002633334 (285h54m20.873716369s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003062191 (285h54m20.874145246s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003211264 (285h54m20.874294299s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003678264 (285h54m20.874761309s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003776632 (285h54m20.874859667s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.004260773 (285h54m20.875343808s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005006633 (285h54m20.876089688s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005886108 (285h54m20.876969153s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.007603749 (285h54m20.878686794s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008077080 (285h54m20.879160115s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008563696 (285h54m20.879646742s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008974920 (285h54m20.880057955s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:22.708545643 (285h54m22.579628688s) [upload=250677]: Next returned nil segment Dec 17 13:23:22.708561813 (285h54m22.579644868s) [upload=250677]: Waiting for error group managing segments... Dec 17 13:23:22.994354835 (285h54m22.86543788s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025599279 (285h54m22.896682325s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025883091 (285h54m22.896966136s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.027657109 (285h54m22.898740154s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.161271635 (285h54m23.03235468s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.169404602 (285h54m23.040487637s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.291383432 (285h54m23.162466487s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.431588217 (285h54m23.302671262s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.507271842 (285h54m23.378354887s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.547288672 (285h54m23.418371717s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.549713660 (285h54m23.420796705s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.061282930 (285h54m23.932365975s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442864742 (285h54m24.313947787s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442915079 (285h54m24.313998124s) [seg_pos={0 0}, upload=250677]: Segment reached optimal threshold of 13 pieces. Dec 17 13:23:24.443325549 (285h54m24.314408594s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443336471 (285h54m24.314419516s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443468612 (285h54m24.314551657s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443488019 (285h54m24.314571054s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443553063 (285h54m24.314636108s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443563332 (285h54m24.314646387s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443620131 (285h54m24.314703176s) [seg_pos={0 0}, upload=250677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:23:24.443771238 (285h54m24.314854293s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:23:24.443779544 (285h54m24.314862579s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:23:24.844328261 (285h54m24.715411316s) [upload=250677]: Done uploading segments... Dec 17 13:23:24.844361535 (285h54m24.71544458s) [upload=250677]: Upload finished. err: <nil> Duration: 5.948850686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.X] duration: 5.001593622s trace: https://jaeger.dev.storj.io/trace/486180babf19d61a FAIL (2025-12-17 13:34:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001617577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....................................................XX.] duration: 28.2820549s trace: https://jaeger.dev.storj.io/trace/93237fddcef4f66 FAIL (2025-12-17 13:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.282082462s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.463685825s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:33:59 UTC) orbiter warning: context deadline exceeded Duration: 24.463716794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.019874025s trace: https://jaeger.dev.storj.io/trace/6aa3dce83f9968cb FAIL (2025-12-17 13:33:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 13:23:20.683120844 (285h54m20.554203889s) [upload=250677]: Starting upload Dec 17 13:23:20.683181839 (285h54m20.554264874s) [upload=250677]: Done starting upload Dec 17 13:23:20.683290607 (285h54m20.554373652s) [upload=250677]: Uploading segments... Dec 17 13:23:20.683694366 (285h54m20.554777411s) [upload=250677]: Got next segment. Inline: false Dec 17 13:23:20.683749731 (285h54m20.554832767s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:23:20.683755452 (285h54m20.554838487s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:23:20.999940787 (285h54m20.871023832s) [seg_pos={0 0}, upload=250677]: Begin upload segment... Dec 17 13:23:21.000001873 (285h54m20.871084938s) [seg_pos={0 0}, upload=250677]: Done begin upload segment. Dec 17 13:23:21.000039866 (285h54m20.871122901s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001756816 (285h54m20.872839861s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001853980 (285h54m20.872937025s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002271615 (285h54m20.87335465s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002633334 (285h54m20.873716369s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003062191 (285h54m20.874145246s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003211264 (285h54m20.874294299s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003678264 (285h54m20.874761309s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003776632 (285h54m20.874859667s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.004260773 (285h54m20.875343808s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005006633 (285h54m20.876089688s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005886108 (285h54m20.876969153s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.007603749 (285h54m20.878686794s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008077080 (285h54m20.879160115s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008563696 (285h54m20.879646742s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008974920 (285h54m20.880057955s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:22.708545643 (285h54m22.579628688s) [upload=250677]: Next returned nil segment Dec 17 13:23:22.708561813 (285h54m22.579644868s) [upload=250677]: Waiting for error group managing segments... Dec 17 13:23:22.994354835 (285h54m22.86543788s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025599279 (285h54m22.896682325s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025883091 (285h54m22.896966136s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.027657109 (285h54m22.898740154s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.161271635 (285h54m23.03235468s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.169404602 (285h54m23.040487637s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.291383432 (285h54m23.162466487s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.431588217 (285h54m23.302671262s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.507271842 (285h54m23.378354887s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.547288672 (285h54m23.418371717s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.549713660 (285h54m23.420796705s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.061282930 (285h54m23.932365975s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442864742 (285h54m24.313947787s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442915079 (285h54m24.313998124s) [seg_pos={0 0}, upload=250677]: Segment reached optimal threshold of 13 pieces. Dec 17 13:23:24.443325549 (285h54m24.314408594s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443336471 (285h54m24.314419516s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443468612 (285h54m24.314551657s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443488019 (285h54m24.314571054s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443553063 (285h54m24.314636108s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443563332 (285h54m24.314646387s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443620131 (285h54m24.314703176s) [seg_pos={0 0}, upload=250677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:23:24.443771238 (285h54m24.314854293s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:23:24.443779544 (285h54m24.314862579s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:23:24.844328261 (285h54m24.715411316s) [upload=250677]: Done uploading segments... Dec 17 13:23:24.844361535 (285h54m24.71544458s) [upload=250677]: Upload finished. err: <nil> Duration: 6.019900665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X..X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX.] duration: 4.205064625s trace: https://jaeger.dev.storj.io/trace/3e525ea4118ed194 FAIL (2025-12-17 13:33:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.205093131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....................................................XX] duration: 30.001363307s trace: https://jaeger.dev.storj.io/trace/93237fddcef4f66 FAIL (2025-12-17 13:33:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00139116s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X..X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.XX] duration: 5.001616102s trace: https://jaeger.dev.storj.io/trace/3e525ea4118ed194 FAIL (2025-12-17 13:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001640279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.883223979s trace: https://jaeger.dev.storj.io/trace/6aa3dce83f9968cb FAIL (2025-12-17 13:32:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 13:23:20.683120844 (285h54m20.554203889s) [upload=250677]: Starting upload Dec 17 13:23:20.683181839 (285h54m20.554264874s) [upload=250677]: Done starting upload Dec 17 13:23:20.683290607 (285h54m20.554373652s) [upload=250677]: Uploading segments... Dec 17 13:23:20.683694366 (285h54m20.554777411s) [upload=250677]: Got next segment. Inline: false Dec 17 13:23:20.683749731 (285h54m20.554832767s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 13:23:20.683755452 (285h54m20.554838487s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 13:23:20.999940787 (285h54m20.871023832s) [seg_pos={0 0}, upload=250677]: Begin upload segment... Dec 17 13:23:21.000001873 (285h54m20.871084938s) [seg_pos={0 0}, upload=250677]: Done begin upload segment. Dec 17 13:23:21.000039866 (285h54m20.871122901s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001756816 (285h54m20.872839861s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.001853980 (285h54m20.872937025s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002271615 (285h54m20.87335465s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.002633334 (285h54m20.873716369s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003062191 (285h54m20.874145246s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003211264 (285h54m20.874294299s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003678264 (285h54m20.874761309s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.003776632 (285h54m20.874859667s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.004260773 (285h54m20.875343808s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005006633 (285h54m20.876089688s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.005886108 (285h54m20.876969153s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.007603749 (285h54m20.878686794s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008077080 (285h54m20.879160115s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008563696 (285h54m20.879646742s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:21.008974920 (285h54m20.880057955s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Uploading piece... Dec 17 13:23:22.708545643 (285h54m22.579628688s) [upload=250677]: Next returned nil segment Dec 17 13:23:22.708561813 (285h54m22.579644868s) [upload=250677]: Waiting for error group managing segments... Dec 17 13:23:22.994354835 (285h54m22.86543788s) [noise=true , address=145.79.227.16:4911 , piece_id=H6TK63WTPYO75YKUVUM6YWSEHSCF5H2YCWFJLGQH4R4KZAT75WTA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025599279 (285h54m22.896682325s) [noise=true , address=145.79.227.40:4050 , piece_id=CIRNGCNO3IBL3LOIQFF2UOZ4OWYT7K66776N46BRMFQFJDIOBBNQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.025883091 (285h54m22.896966136s) [noise=true , address=145.79.227.51:4409 , piece_id=KWKN67EHVCDB5UAMKJWV5EOPQEJFA3QOUAQJFFYVC6XK23EP26ZA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.027657109 (285h54m22.898740154s) [noise=true , address=145.79.227.39:4013 , piece_id=MGMDGEOPSU7PX7EUH4IBGFF76MOXRKK2DOBHBB4ICSDTDJGB6XJA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.161271635 (285h54m23.03235468s) [noise=true , address=145.79.227.32:4835 , piece_id=AN66ACF4KCJSF3UVHJ6OH6W5GUPNRUYUHTWKCO3OUE3ZNTPROPIQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.169404602 (285h54m23.040487637s) [noise=true , address=145.79.227.41:4087 , piece_id=O4LMO34XSKXL5YO5ZNZ5SFKAA3NKEWLTKHCCXGFD2JXMW2UQPSPA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.291383432 (285h54m23.162466487s) [noise=true , address=77.90.31.14:4664 , piece_id=UCWRXSIUWM2XG4YFRJRCIBRE5FZIGAFMHYJDUXRVMQ5UKXC6QYEQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.431588217 (285h54m23.302671262s) [noise=true , address=45.91.116.14:4192 , piece_id=KQY34EMVKQM6GNAIMTRJVY67NFQMXQHMXLTP763BL74I2HPXOVKA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.507271842 (285h54m23.378354887s) [noise=true , address=145.79.227.21:4474 , piece_id=XSZCAKVKY4WOYXPTSZX6GYX6EC4KLNGYNDWEPA4QJVZCPIYLD3QA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.547288672 (285h54m23.418371717s) [noise=true , address=145.79.227.36:4519 , piece_id=XS6RXLOMKDUDEKQYPITQO35VK6AZW7RE6A3GZE27C6K6RFAQ6BIA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:23.549713660 (285h54m23.420796705s) [noise=true , address=145.79.227.46:4601 , piece_id=FIVR7RUD56UBTNJYE3B3WVXG5SJQWKUKGXHSP65IEBUXZSNG5BMQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.061282930 (285h54m23.932365975s) [noise=true , address=145.79.227.18:4006 , piece_id=YGJVPAFCYUPGTYVGW7VD7GOKV3DKQBWT7GOKBZLEEWIXGVRDOGGQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442864742 (285h54m24.313947787s) [noise=true , address=45.91.116.6:4974 , piece_id=IPM3BUOH7I2NR7TQFHXT3DLFMATTQQF2QHDDZN6MZWPUVSWKSSDA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: <nil> Dec 17 13:23:24.442915079 (285h54m24.313998124s) [seg_pos={0 0}, upload=250677]: Segment reached optimal threshold of 13 pieces. Dec 17 13:23:24.443325549 (285h54m24.314408594s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443336471 (285h54m24.314419516s) [noise=true , address=145.79.227.43:4159 , piece_id=JUZLCWBFNUU2KM3NVDNRDFM6MUSOSCP4HOWLISJYHGLLLR36AWQA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443468612 (285h54m24.314551657s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443488019 (285h54m24.314571054s) [noise=true , address=145.79.227.33:4871 , piece_id=6JXVZ65BKZZDD5472RLS2MKCE2ILGEFYVMZPGS3NWO7U3GKVI4JA, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443553063 (285h54m24.314636108s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 17 13:23:24.443563332 (285h54m24.314646387s) [noise=true , address=145.79.227.52:4445 , piece_id=5UANANJPBDAIPV3WABV2C732FK3BSH3E3IXFTDZH6EMQIWAWGCOQ, seg_pos={0 0}, upload=250677]: Long tail cancellation due to optimal threshold Dec 17 13:23:24.443620131 (285h54m24.314703176s) [seg_pos={0 0}, upload=250677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 13:23:24.443771238 (285h54m24.314854293s) [upload=250677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 13:23:24.443779544 (285h54m24.314862579s) [upload=250677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 13:23:24.844328261 (285h54m24.715411316s) [upload=250677]: Done uploading segments... Dec 17 13:23:24.844361535 (285h54m24.71544458s) [upload=250677]: Upload finished. err: <nil> Duration: 5.883248616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.410422997s trace: https://jaeger.dev.storj.io/trace/6397e2ef55f83406 FAIL (2025-12-17 13:32:29 UTC) orbiter warning: context deadline exceeded Duration: 7.410456912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X..X.....................XX......XX.XXX.XX.XXXXX..XXXXXX.X] duration: 5.001965287s trace: https://jaeger.dev.storj.io/trace/71ba12cd80e70f30 FAIL (2025-12-17 13:32:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002002449s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)