Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 861992 reports (Page 11740 of 17240)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..............................X....................X.] duration: 5.538283198s trace: https://jaeger.dev.storj.io/trace/10deaed4f0583195 FAIL (2025-12-08 07:03:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.538304128s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..............................X....................X] duration: 3m54.084459253s trace: https://jaeger.dev.storj.io/trace/10deaed4f0583195 FAIL (2025-12-08 07:03:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3m54.084483568s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...........................X.X......X................] duration: 5.656200154s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 07:03:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.656231464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X.................................X................] duration: 3.589824956s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 07:03:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589856736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........................X.X......X...............] duration: 5.777288972s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 07:02:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.777357562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X.................................X...............] duration: 3.420065279s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 07:02:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420093453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...........................X.X......X..............] duration: 5.911795797s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 07:02:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.911829502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X.................................X..............] duration: 4.331082612s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 07:01:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.33113359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X.................................X.............] duration: 3.408931156s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 07:01:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.409170491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.458754443s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 07:01:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.458782096s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........................X.X......X.............] duration: 5.849790999s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 07:01:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.84981768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.218919868s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 07:01:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.21894658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.441313645s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 07:00:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.441346768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...........................X.X......X............] duration: 5.897275749s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 07:00:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.897363187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X.................................X............] duration: 3.435670965s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 07:00:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435706484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X.................................X...........] duration: 3.385895918s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.385971953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.824935958s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:59:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.824961446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X...........................X.X......X...........] duration: 5.722595663s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:59:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.722625801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.................................X..........] duration: 3.460568503s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:59:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460622105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...........................X.X......X..........] duration: 5.978298832s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:59:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.978326946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.763472563s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:58:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.763499704s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.................................X.........] duration: 3.507620912s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507646661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............................X....................] duration: 5.83917849s trace: https://jaeger.dev.storj.io/trace/16d204ff91e77175 FAIL (2025-12-08 06:58:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.839227124s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...........................X.X......X.........] duration: 5.869909693s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:58:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86993924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.671866609s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:58:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.671889102s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.................................X........] duration: 3.576556059s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:57:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576583331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...........................X.X......X........] duration: 5.850704926s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:57:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.850731848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.738136426s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:57:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.738166062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............................X...................] duration: 5.792547906s trace: https://jaeger.dev.storj.io/trace/16d204ff91e77175 FAIL (2025-12-08 06:57:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.792590307s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...........................X.X......X.......] duration: 5.970813754s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:57:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.970844132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.................................X.......] duration: 3.428127395s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:57:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428157843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............................X..................] duration: 5.903569811s trace: https://jaeger.dev.storj.io/trace/16d204ff91e77175 FAIL (2025-12-08 06:57:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.903600019s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.................................X......] duration: 3.505494036s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:57:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505523873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...........................X.X......X......] duration: 5.579037928s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:56:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.579110906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.025086961s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:56:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.02511329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.................................X.....] duration: 3.367686315s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:56:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367709629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............................X.................] duration: 5.795220135s trace: https://jaeger.dev.storj.io/trace/16d204ff91e77175 FAIL (2025-12-08 06:56:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.795252227s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 476.860241ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/41e297919dd69fc2 FAIL (2025-12-08 06:56:03 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.993786ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.045075831s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:55:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.045113533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X...........................X.X......X.....] duration: 5.666651628s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:55:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.666701203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X.................................X....] duration: 3.389849458s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:55:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389876881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X.................................X...] duration: 3.483290996s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:55:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483318038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............................X................] duration: 5.917229406s trace: https://jaeger.dev.storj.io/trace/16d204ff91e77175 FAIL (2025-12-08 06:55:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.91726322s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X...........................X.X......X....] duration: 5.888662756s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:55:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.888693455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X.................................X..] duration: 3.398281174s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:55:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398460524s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 491.941729ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/41e297919dd69fc2 FAIL (2025-12-08 06:55:03 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 491.97375ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.876065679s trace: https://jaeger.dev.storj.io/trace/2313adc163c8e057 FAIL (2025-12-08 06:54:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments... Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment... Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment. Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece... Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments... Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil> Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces. Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments... Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.876094553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X.................................X.] duration: 3.452928571s trace: https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0 FAIL (2025-12-08 06:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452963748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X...........................X.X......X...] duration: 5.833955262s trace: https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb FAIL (2025-12-08 06:54:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.83398065s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 478.060877ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/41e297919dd69fc2 FAIL (2025-12-08 06:54:24 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 478.084913ms core satellite eu1

1
...
11738
11739
11740
11741
11742
...
17240