Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.516744566s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:08:15 UTC) orbiter warning: context deadline exceeded Duration: 3.516767019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X.] duration: 7.052989103s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:07:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 7.053007798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X] duration: 30.002541904s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:07:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 30.002563476s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.425485202s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:07:11 UTC) orbiter warning: context deadline exceeded Duration: 3.42552085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....] duration: 6.045468522s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 6.045484194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X...] duration: 26.306514152s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 26.306549269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.531660249s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:06:11 UTC) orbiter warning: context deadline exceeded Duration: 3.531690128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.542838604s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:05:50 UTC) orbiter warning: context deadline exceeded Duration: 3.542862509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X..] duration: 7.365688378s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 7.365723345s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.430921934s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:04:54 UTC) orbiter warning: context deadline exceeded Duration: 3.431072371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.492130235s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:04:38 UTC) orbiter warning: context deadline exceeded Duration: 3.492155843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.544017734s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:04:20 UTC) orbiter warning: context deadline exceeded Duration: 3.544050235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X.] duration: 22.365861473s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 22.365879227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.49893213s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:03:34 UTC) orbiter warning: context deadline exceeded Duration: 3.498958079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X] duration: 30.003582827s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:03:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 30.003606832s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.527452595s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:02:41 UTC) orbiter warning: context deadline exceeded Duration: 3.527508612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.527702148s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:02:03 UTC) orbiter warning: context deadline exceeded Duration: 3.5277293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...] duration: 5.939052123s trace: https://jaeger.dev.storj.io/trace/2042579170a79a7 FAIL (2025-12-28 00:01:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:59:27.435359167 (536h30m27.306442222s) [upload=470522]: Starting upload Dec 27 23:59:27.435451884 (536h30m27.306534949s) [upload=470522]: Done starting upload Dec 27 23:59:27.435550592 (536h30m27.306633647s) [upload=470522]: Uploading segments... Dec 27 23:59:27.435882674 (536h30m27.306965719s) [upload=470522]: Got next segment. Inline: false Dec 27 23:59:27.435938661 (536h30m27.307021696s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:59:27.435960422 (536h30m27.307043457s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:59:27.682036529 (536h30m27.553119574s) [seg_pos={0 0}, upload=470522]: Begin upload segment... Dec 27 23:59:27.682077518 (536h30m27.553160553s) [seg_pos={0 0}, upload=470522]: Done begin upload segment. Dec 27 23:59:27.682101373 (536h30m27.553184408s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682109910 (536h30m27.553192945s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682652683 (536h30m27.553735728s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.686991265 (536h30m27.5580743s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.687050066 (536h30m27.558133101s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688193303 (536h30m27.559276338s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688331696 (536h30m27.559414732s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689489141 (536h30m27.560572176s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689516422 (536h30m27.560599457s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689520911 (536h30m27.560603946s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689524749 (536h30m27.560607794s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689528226 (536h30m27.560611261s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689531963 (536h30m27.560615008s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689535669 (536h30m27.560618705s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689539206 (536h30m27.560622241s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689779584 (536h30m27.560862619s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:29.400266945 (536h30m29.27134998s) [upload=470522]: Next returned nil segment Dec 27 23:59:29.400281684 (536h30m29.271364729s) [upload=470522]: Waiting for error group managing segments... Dec 27 23:59:29.683285937 (536h30m29.554368972s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.684355504 (536h30m29.555438539s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.704675446 (536h30m29.575758481s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.719111384 (536h30m29.590194429s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.721221763 (536h30m29.592304798s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.856523279 (536h30m29.727606314s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.876127418 (536h30m29.747210453s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.885257262 (536h30m29.756340297s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.925917405 (536h30m29.79700044s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.935665376 (536h30m29.806748431s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:30.913852196 (536h30m30.784935231s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.255577673 (536h30m31.126660708s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348539617 (536h30m31.219622663s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348558975 (536h30m31.21964202s) [seg_pos={0 0}, upload=470522]: Segment reached optimal threshold of 13 pieces. Dec 27 23:59:31.348755498 (536h30m31.219838543s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348764326 (536h30m31.219847371s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.348965939 (536h30m31.220049004s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348986879 (536h30m31.220069944s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349006046 (536h30m31.220089101s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.349017397 (536h30m31.220100453s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349097289 (536h30m31.220180334s) [seg_pos={0 0}, upload=470522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:59:31.349170889 (536h30m31.220253924s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:59:31.349175639 (536h30m31.220258684s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:59:31.639426531 (536h30m31.510509566s) [upload=470522]: Done uploading segments... Dec 27 23:59:31.639447852 (536h30m31.510530887s) [upload=470522]: Upload finished. err: <nil> Duration: 5.939140922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.462582813s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:01:32 UTC) orbiter warning: context deadline exceeded Duration: 3.462612531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X..] duration: 6.425362327s trace: https://jaeger.dev.storj.io/trace/2042579170a79a7 FAIL (2025-12-28 00:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:59:27.435359167 (536h30m27.306442222s) [upload=470522]: Starting upload Dec 27 23:59:27.435451884 (536h30m27.306534949s) [upload=470522]: Done starting upload Dec 27 23:59:27.435550592 (536h30m27.306633647s) [upload=470522]: Uploading segments... Dec 27 23:59:27.435882674 (536h30m27.306965719s) [upload=470522]: Got next segment. Inline: false Dec 27 23:59:27.435938661 (536h30m27.307021696s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:59:27.435960422 (536h30m27.307043457s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:59:27.682036529 (536h30m27.553119574s) [seg_pos={0 0}, upload=470522]: Begin upload segment... Dec 27 23:59:27.682077518 (536h30m27.553160553s) [seg_pos={0 0}, upload=470522]: Done begin upload segment. Dec 27 23:59:27.682101373 (536h30m27.553184408s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682109910 (536h30m27.553192945s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682652683 (536h30m27.553735728s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.686991265 (536h30m27.5580743s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.687050066 (536h30m27.558133101s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688193303 (536h30m27.559276338s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688331696 (536h30m27.559414732s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689489141 (536h30m27.560572176s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689516422 (536h30m27.560599457s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689520911 (536h30m27.560603946s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689524749 (536h30m27.560607794s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689528226 (536h30m27.560611261s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689531963 (536h30m27.560615008s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689535669 (536h30m27.560618705s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689539206 (536h30m27.560622241s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689779584 (536h30m27.560862619s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:29.400266945 (536h30m29.27134998s) [upload=470522]: Next returned nil segment Dec 27 23:59:29.400281684 (536h30m29.271364729s) [upload=470522]: Waiting for error group managing segments... Dec 27 23:59:29.683285937 (536h30m29.554368972s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.684355504 (536h30m29.555438539s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.704675446 (536h30m29.575758481s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.719111384 (536h30m29.590194429s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.721221763 (536h30m29.592304798s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.856523279 (536h30m29.727606314s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.876127418 (536h30m29.747210453s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.885257262 (536h30m29.756340297s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.925917405 (536h30m29.79700044s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.935665376 (536h30m29.806748431s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:30.913852196 (536h30m30.784935231s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.255577673 (536h30m31.126660708s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348539617 (536h30m31.219622663s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348558975 (536h30m31.21964202s) [seg_pos={0 0}, upload=470522]: Segment reached optimal threshold of 13 pieces. Dec 27 23:59:31.348755498 (536h30m31.219838543s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348764326 (536h30m31.219847371s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.348965939 (536h30m31.220049004s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348986879 (536h30m31.220069944s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349006046 (536h30m31.220089101s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.349017397 (536h30m31.220100453s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349097289 (536h30m31.220180334s) [seg_pos={0 0}, upload=470522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:59:31.349170889 (536h30m31.220253924s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:59:31.349175639 (536h30m31.220258684s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:59:31.639426531 (536h30m31.510509566s) [upload=470522]: Done uploading segments... Dec 27 23:59:31.639447852 (536h30m31.510530887s) [upload=470522]: Upload finished. err: <nil> Duration: 6.42539479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.527544503s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:01:14 UTC) orbiter warning: context deadline exceeded Duration: 3.527570651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.623601409s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:00:39 UTC) orbiter warning: context deadline exceeded Duration: 3.623632658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [...XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X.] duration: 6.202614589s trace: https://jaeger.dev.storj.io/trace/2042579170a79a7 FAIL (2025-12-28 00:00:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:59:27.435359167 (536h30m27.306442222s) [upload=470522]: Starting upload Dec 27 23:59:27.435451884 (536h30m27.306534949s) [upload=470522]: Done starting upload Dec 27 23:59:27.435550592 (536h30m27.306633647s) [upload=470522]: Uploading segments... Dec 27 23:59:27.435882674 (536h30m27.306965719s) [upload=470522]: Got next segment. Inline: false Dec 27 23:59:27.435938661 (536h30m27.307021696s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:59:27.435960422 (536h30m27.307043457s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:59:27.682036529 (536h30m27.553119574s) [seg_pos={0 0}, upload=470522]: Begin upload segment... Dec 27 23:59:27.682077518 (536h30m27.553160553s) [seg_pos={0 0}, upload=470522]: Done begin upload segment. Dec 27 23:59:27.682101373 (536h30m27.553184408s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682109910 (536h30m27.553192945s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682652683 (536h30m27.553735728s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.686991265 (536h30m27.5580743s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.687050066 (536h30m27.558133101s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688193303 (536h30m27.559276338s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688331696 (536h30m27.559414732s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689489141 (536h30m27.560572176s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689516422 (536h30m27.560599457s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689520911 (536h30m27.560603946s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689524749 (536h30m27.560607794s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689528226 (536h30m27.560611261s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689531963 (536h30m27.560615008s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689535669 (536h30m27.560618705s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689539206 (536h30m27.560622241s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689779584 (536h30m27.560862619s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:29.400266945 (536h30m29.27134998s) [upload=470522]: Next returned nil segment Dec 27 23:59:29.400281684 (536h30m29.271364729s) [upload=470522]: Waiting for error group managing segments... Dec 27 23:59:29.683285937 (536h30m29.554368972s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.684355504 (536h30m29.555438539s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.704675446 (536h30m29.575758481s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.719111384 (536h30m29.590194429s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.721221763 (536h30m29.592304798s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.856523279 (536h30m29.727606314s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.876127418 (536h30m29.747210453s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.885257262 (536h30m29.756340297s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.925917405 (536h30m29.79700044s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.935665376 (536h30m29.806748431s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:30.913852196 (536h30m30.784935231s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.255577673 (536h30m31.126660708s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348539617 (536h30m31.219622663s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348558975 (536h30m31.21964202s) [seg_pos={0 0}, upload=470522]: Segment reached optimal threshold of 13 pieces. Dec 27 23:59:31.348755498 (536h30m31.219838543s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348764326 (536h30m31.219847371s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.348965939 (536h30m31.220049004s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348986879 (536h30m31.220069944s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349006046 (536h30m31.220089101s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.349017397 (536h30m31.220100453s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349097289 (536h30m31.220180334s) [seg_pos={0 0}, upload=470522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:59:31.349170889 (536h30m31.220253924s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:59:31.349175639 (536h30m31.220258684s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:59:31.639426531 (536h30m31.510509566s) [upload=470522]: Done uploading segments... Dec 27 23:59:31.639447852 (536h30m31.510530887s) [upload=470522]: Upload finished. err: <nil> Duration: 6.202633544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [....XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X] duration: 30.002347774s trace: https://jaeger.dev.storj.io/trace/2042579170a79a7 FAIL (2025-12-27 23:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:59:27.435359167 (536h30m27.306442222s) [upload=470522]: Starting upload Dec 27 23:59:27.435451884 (536h30m27.306534949s) [upload=470522]: Done starting upload Dec 27 23:59:27.435550592 (536h30m27.306633647s) [upload=470522]: Uploading segments... Dec 27 23:59:27.435882674 (536h30m27.306965719s) [upload=470522]: Got next segment. Inline: false Dec 27 23:59:27.435938661 (536h30m27.307021696s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:59:27.435960422 (536h30m27.307043457s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:59:27.682036529 (536h30m27.553119574s) [seg_pos={0 0}, upload=470522]: Begin upload segment... Dec 27 23:59:27.682077518 (536h30m27.553160553s) [seg_pos={0 0}, upload=470522]: Done begin upload segment. Dec 27 23:59:27.682101373 (536h30m27.553184408s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682109910 (536h30m27.553192945s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.682652683 (536h30m27.553735728s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.686991265 (536h30m27.5580743s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.687050066 (536h30m27.558133101s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688193303 (536h30m27.559276338s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.688331696 (536h30m27.559414732s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689489141 (536h30m27.560572176s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689516422 (536h30m27.560599457s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689520911 (536h30m27.560603946s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689524749 (536h30m27.560607794s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689528226 (536h30m27.560611261s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689531963 (536h30m27.560615008s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689535669 (536h30m27.560618705s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689539206 (536h30m27.560622241s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:27.689779584 (536h30m27.560862619s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Uploading piece... Dec 27 23:59:29.400266945 (536h30m29.27134998s) [upload=470522]: Next returned nil segment Dec 27 23:59:29.400281684 (536h30m29.271364729s) [upload=470522]: Waiting for error group managing segments... Dec 27 23:59:29.683285937 (536h30m29.554368972s) [noise=true , address=145.79.227.36:4519 , piece_id=PAKSUFNZXCF5IRDDLYA67D5F46HB63NWYNMMI4L63M5J475A7OCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.684355504 (536h30m29.555438539s) [noise=true , address=145.79.227.9:4367 , piece_id=NPCXOKIIL2V2VY4D3OPGXZXT2YFH3T6E6ZTP5HKRA4TPRBJKIGDA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.704675446 (536h30m29.575758481s) [noise=true , address=145.79.227.47:4265 , piece_id=XYEQAGUOPFZB545JI27FR75ORXC7TPHC4LVCH4DHBPDVDPDOUTSQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.719111384 (536h30m29.590194429s) [noise=true , address=145.79.227.15:4874 , piece_id=ATJZX3M65ZSESMLY33FRDKOLX7CZTRHVDNBO4IX4K6HNUAC7D5RA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.721221763 (536h30m29.592304798s) [noise=true , address=145.79.227.17:4971 , piece_id=GLMDNXN2CR6QT7JI4U2HGHLBX23KL5HEHC4KJHAZQ6H6OJUJQKRQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.856523279 (536h30m29.727606314s) [noise=true , address=77.90.31.16:4736 , piece_id=F7RHXH5NFPZSD6SFA3G6NAVXWCIXCNFGMVQCXVBKDCPADDT2INCA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.876127418 (536h30m29.747210453s) [noise=true , address=77.90.31.14:4661 , piece_id=NFUULYBJIWSVPEYVNJ3SYMGZ3D64SHCDZ3AAVNGLNRWUTDCDUI5A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.885257262 (536h30m29.756340297s) [noise=true , address=77.90.31.17:4773 , piece_id=FWFZXMLVWERR5D7OOCDOXH2IMD7UMKWAKNYD3DNBMD7XPUG2BA6A, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.925917405 (536h30m29.79700044s) [noise=true , address=77.90.31.18:4808 , piece_id=WJ32SYWSB7HSMEYCUJEHBVYU7SZLRMZGZTXGU62RCBNQNTKDU74Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:29.935665376 (536h30m29.806748431s) [noise=true , address=77.90.31.13:4626 , piece_id=YMFFIR3G4AZRDATODZTRZAS52GJIS33KWWMI6DTNQUX3BFUGYHMA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:30.913852196 (536h30m30.784935231s) [noise=true , address=145.79.227.44:4193 , piece_id=WQWHE2L6B3XTN2HL6AENDOMWBKVIZ6LVLYUP4RU22U3PBE4HNC4Q, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.255577673 (536h30m31.126660708s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FJRYARNIVLSJHK5C4J6W56MTHMEMKL5ATUZCWVYY3BI5USNHWQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348539617 (536h30m31.219622663s) [noise=true , address=45.91.116.4:4939 , piece_id=6N5657KI2FQDB773CWE7POZTNQZOYAACBPXUEOLEJRCEZKE54SYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: <nil> Dec 27 23:59:31.348558975 (536h30m31.21964202s) [seg_pos={0 0}, upload=470522]: Segment reached optimal threshold of 13 pieces. Dec 27 23:59:31.348755498 (536h30m31.219838543s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348764326 (536h30m31.219847371s) [noise=true , address=145.79.227.49:4338 , piece_id=6ESO3KLQGZNOQCD4GSY7M4R6C5VZD4RLG6AKY7RFN3U5TISH6KFQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.348965939 (536h30m31.220049004s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.348986879 (536h30m31.220069944s) [noise=true , address=145.79.227.3:4150 , piece_id=F4VIIW4MZKWLUU57WINXJOVMTQV4XNBJSEJWRXI66G5TNDHHG2XA, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349006046 (536h30m31.220089101s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:59:31.349017397 (536h30m31.220100453s) [noise=true , address=145.79.227.33:4870 , piece_id=LPPIJFTJDAJ7RAPZILUDFDBN3W3FMRVAJBPFBWYOZFPYXABQZCYQ, seg_pos={0 0}, upload=470522]: Long tail cancellation due to optimal threshold Dec 27 23:59:31.349097289 (536h30m31.220180334s) [seg_pos={0 0}, upload=470522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:59:31.349170889 (536h30m31.220253924s) [upload=470522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:59:31.349175639 (536h30m31.220258684s) [upload=470522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:59:31.639426531 (536h30m31.510509566s) [upload=470522]: Done uploading segments... Dec 27 23:59:31.639447852 (536h30m31.510530887s) [upload=470522]: Upload finished. err: <nil> Duration: 30.002370387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.481288179s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:59:34 UTC) orbiter warning: context deadline exceeded Duration: 3.48131519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.454006111s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:59:08 UTC) orbiter warning: context deadline exceeded Duration: 3.454046628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.....XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.] duration: 8.410186385s trace: https://jaeger.dev.storj.io/trace/6d47fc051ad9aad4 FAIL (2025-12-27 23:58:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:57:25.328010354 (536h28m25.199093399s) [upload=470490]: Starting upload Dec 27 23:57:25.328128288 (536h28m25.199211334s) [upload=470490]: Done starting upload Dec 27 23:57:25.328207008 (536h28m25.199290043s) [upload=470490]: Uploading segments... Dec 27 23:57:25.328493183 (536h28m25.199576238s) [upload=470490]: Got next segment. Inline: false Dec 27 23:57:25.328549561 (536h28m25.199632606s) [upload=470490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:57:25.328555762 (536h28m25.199638817s) [upload=470490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:57:25.576095888 (536h28m25.447178933s) [seg_pos={0 0}, upload=470490]: Begin upload segment... Dec 27 23:57:25.576145031 (536h28m25.447228077s) [seg_pos={0 0}, upload=470490]: Done begin upload segment. Dec 27 23:57:25.576178485 (536h28m25.44726153s) [noise=true , address=45.91.116.5:4297 , piece_id=YCX6OBBLRR3N4G7GY5NA3CISZRZ7M3QKOJ4M6LEYDTTDIBTCV6CQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.576198454 (536h28m25.447281499s) [noise=true , address=145.79.227.15:4873 , piece_id=2GDEG74PPW63ZZKPBG3KHSSEZP2EM7VM7ZXJUCI5LRXNUGTNLNTQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.576670682 (536h28m25.447753727s) [noise=true , address=45.91.116.10:4083 , piece_id=27K2222F2VAVO3U7D6RK4HRSQ7V4UKFFQZRKM7MQ5F37UDHI7IHA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577081014 (536h28m25.448164059s) [noise=true , address=145.79.227.9:4367 , piece_id=SUH2FLIOESG4G77L7RJHCROCDAH5VQS627HMQRGV337GI42NNPDQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577476357 (536h28m25.448559402s) [noise=true , address=45.91.116.12:4227 , piece_id=BE3XNZOPIZWQXF65HFS44UWVFGCXJLRNVLKIOVTA4X5E6L2IMEXQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577906947 (536h28m25.448989992s) [noise=true , address=145.79.227.17:4972 , piece_id=SEQBN337U6YRTR4DGZIBKYOL465T5RPLX5SG32RUTACGS2PAOO7Q, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578399454 (536h28m25.44948249s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578417870 (536h28m25.449500915s) [noise=true , address=77.90.31.14:4661 , piece_id=FWPDKOP77HVUG5XLSEJEOAGPCD42P5QONM3BWHU747VG6DFCJHUA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578736366 (536h28m25.449819411s) [noise=true , address=45.91.116.9:4261 , piece_id=D2GEEL7H3YBOIYAZVMN62XFWSH2FTTACN56EECHFISFTEGY5AIQQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579072315 (536h28m25.45015535s) [noise=true , address=145.79.227.45:4231 , piece_id=SAEUUUODIGOA4JMJSM34YQWELB3BK2DSXIX7O5D2OF23F2HMDGHQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579327672 (536h28m25.450410717s) [noise=true , address=45.91.116.14:4191 , piece_id=YVLTDV3GZMCGGRZUJACGPAXAQT5BFJ5I6VYTL3DJO5QKV275VK5A, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579666077 (536h28m25.450749112s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.580071989 (536h28m25.451155024s) [noise=true , address=145.79.227.7:4295 , piece_id=ZDQ6IT3V7KH4QR67V5CJM2G5A4MTKHW3CPHD2OCYPK7I3Z6ECEZA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.580393712 (536h28m25.451476747s) [noise=true , address=145.79.227.52:4445 , piece_id=B2GIXSUAQASWCPU2C2WD5ZQPXX4JXQVYNCEBPB3TCONGPU6DXPVA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.589671408 (536h28m25.460754454s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.596618467 (536h28m25.467701512s) [noise=true , address=45.91.116.7:4009 , piece_id=PM2PRTZVIIFRSURNFEJWFQH2F6IVSQUDVHI22TXBZ32X4CS4Y7NQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:27.288823307 (536h28m27.159906352s) [upload=470490]: Next returned nil segment Dec 27 23:57:27.288839587 (536h28m27.159922633s) [upload=470490]: Waiting for error group managing segments... Dec 27 23:57:27.590060449 (536h28m27.461143484s) [noise=true , address=145.79.227.17:4972 , piece_id=SEQBN337U6YRTR4DGZIBKYOL465T5RPLX5SG32RUTACGS2PAOO7Q, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.590570340 (536h28m27.461653375s) [noise=true , address=145.79.227.45:4231 , piece_id=SAEUUUODIGOA4JMJSM34YQWELB3BK2DSXIX7O5D2OF23F2HMDGHQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.598426689 (536h28m27.469509724s) [noise=true , address=145.79.227.52:4445 , piece_id=B2GIXSUAQASWCPU2C2WD5ZQPXX4JXQVYNCEBPB3TCONGPU6DXPVA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.598906132 (536h28m27.469989168s) [noise=true , address=145.79.227.7:4295 , piece_id=ZDQ6IT3V7KH4QR67V5CJM2G5A4MTKHW3CPHD2OCYPK7I3Z6ECEZA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.600271612 (536h28m27.471354647s) [noise=true , address=145.79.227.9:4367 , piece_id=SUH2FLIOESG4G77L7RJHCROCDAH5VQS627HMQRGV337GI42NNPDQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.605502683 (536h28m27.476585728s) [noise=true , address=145.79.227.15:4873 , piece_id=2GDEG74PPW63ZZKPBG3KHSSEZP2EM7VM7ZXJUCI5LRXNUGTNLNTQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.811362122 (536h28m27.682445157s) [noise=true , address=45.91.116.12:4227 , piece_id=BE3XNZOPIZWQXF65HFS44UWVFGCXJLRNVLKIOVTA4X5E6L2IMEXQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.814354940 (536h28m27.685437975s) [noise=true , address=77.90.31.14:4661 , piece_id=FWPDKOP77HVUG5XLSEJEOAGPCD42P5QONM3BWHU747VG6DFCJHUA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.831121607 (536h28m27.702204652s) [noise=true , address=45.91.116.9:4261 , piece_id=D2GEEL7H3YBOIYAZVMN62XFWSH2FTTACN56EECHFISFTEGY5AIQQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.837465406 (536h28m27.708548442s) [noise=true , address=45.91.116.5:4297 , piece_id=YCX6OBBLRR3N4G7GY5NA3CISZRZ7M3QKOJ4M6LEYDTTDIBTCV6CQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.840602800 (536h28m27.711685846s) [noise=true , address=45.91.116.14:4191 , piece_id=YVLTDV3GZMCGGRZUJACGPAXAQT5BFJ5I6VYTL3DJO5QKV275VK5A, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.840961353 (536h28m27.712044398s) [noise=true , address=45.91.116.10:4083 , piece_id=27K2222F2VAVO3U7D6RK4HRSQ7V4UKFFQZRKM7MQ5F37UDHI7IHA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.881586911 (536h28m27.752669946s) [noise=true , address=45.91.116.7:4009 , piece_id=PM2PRTZVIIFRSURNFEJWFQH2F6IVSQUDVHI22TXBZ32X4CS4Y7NQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.881631105 (536h28m27.75271414s) [seg_pos={0 0}, upload=470490]: Segment reached optimal threshold of 13 pieces. Dec 27 23:57:27.881919604 (536h28m27.753002659s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.881929853 (536h28m27.753012898s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882088035 (536h28m27.75317107s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.882108905 (536h28m27.75319195s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882337460 (536h28m27.753420505s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.882348260 (536h28m27.753431305s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882375522 (536h28m27.753458557s) [seg_pos={0 0}, upload=470490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:57:27.882466726 (536h28m27.753549771s) [upload=470490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:57:27.882473569 (536h28m27.753556604s) [upload=470490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:57:28.178184988 (536h28m28.049268023s) [upload=470490]: Done uploading segments... Dec 27 23:57:28.178203413 (536h28m28.049286448s) [upload=470490]: Upload finished. err: <nil> Duration: 8.410213156s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 4.170494556s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:58:19 UTC) orbiter warning: context deadline exceeded Duration: 4.17052226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [......XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X] duration: 30.002115298s trace: https://jaeger.dev.storj.io/trace/6d47fc051ad9aad4 FAIL (2025-12-27 23:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:57:25.328010354 (536h28m25.199093399s) [upload=470490]: Starting upload Dec 27 23:57:25.328128288 (536h28m25.199211334s) [upload=470490]: Done starting upload Dec 27 23:57:25.328207008 (536h28m25.199290043s) [upload=470490]: Uploading segments... Dec 27 23:57:25.328493183 (536h28m25.199576238s) [upload=470490]: Got next segment. Inline: false Dec 27 23:57:25.328549561 (536h28m25.199632606s) [upload=470490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:57:25.328555762 (536h28m25.199638817s) [upload=470490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:57:25.576095888 (536h28m25.447178933s) [seg_pos={0 0}, upload=470490]: Begin upload segment... Dec 27 23:57:25.576145031 (536h28m25.447228077s) [seg_pos={0 0}, upload=470490]: Done begin upload segment. Dec 27 23:57:25.576178485 (536h28m25.44726153s) [noise=true , address=45.91.116.5:4297 , piece_id=YCX6OBBLRR3N4G7GY5NA3CISZRZ7M3QKOJ4M6LEYDTTDIBTCV6CQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.576198454 (536h28m25.447281499s) [noise=true , address=145.79.227.15:4873 , piece_id=2GDEG74PPW63ZZKPBG3KHSSEZP2EM7VM7ZXJUCI5LRXNUGTNLNTQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.576670682 (536h28m25.447753727s) [noise=true , address=45.91.116.10:4083 , piece_id=27K2222F2VAVO3U7D6RK4HRSQ7V4UKFFQZRKM7MQ5F37UDHI7IHA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577081014 (536h28m25.448164059s) [noise=true , address=145.79.227.9:4367 , piece_id=SUH2FLIOESG4G77L7RJHCROCDAH5VQS627HMQRGV337GI42NNPDQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577476357 (536h28m25.448559402s) [noise=true , address=45.91.116.12:4227 , piece_id=BE3XNZOPIZWQXF65HFS44UWVFGCXJLRNVLKIOVTA4X5E6L2IMEXQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.577906947 (536h28m25.448989992s) [noise=true , address=145.79.227.17:4972 , piece_id=SEQBN337U6YRTR4DGZIBKYOL465T5RPLX5SG32RUTACGS2PAOO7Q, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578399454 (536h28m25.44948249s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578417870 (536h28m25.449500915s) [noise=true , address=77.90.31.14:4661 , piece_id=FWPDKOP77HVUG5XLSEJEOAGPCD42P5QONM3BWHU747VG6DFCJHUA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.578736366 (536h28m25.449819411s) [noise=true , address=45.91.116.9:4261 , piece_id=D2GEEL7H3YBOIYAZVMN62XFWSH2FTTACN56EECHFISFTEGY5AIQQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579072315 (536h28m25.45015535s) [noise=true , address=145.79.227.45:4231 , piece_id=SAEUUUODIGOA4JMJSM34YQWELB3BK2DSXIX7O5D2OF23F2HMDGHQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579327672 (536h28m25.450410717s) [noise=true , address=45.91.116.14:4191 , piece_id=YVLTDV3GZMCGGRZUJACGPAXAQT5BFJ5I6VYTL3DJO5QKV275VK5A, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.579666077 (536h28m25.450749112s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.580071989 (536h28m25.451155024s) [noise=true , address=145.79.227.7:4295 , piece_id=ZDQ6IT3V7KH4QR67V5CJM2G5A4MTKHW3CPHD2OCYPK7I3Z6ECEZA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.580393712 (536h28m25.451476747s) [noise=true , address=145.79.227.52:4445 , piece_id=B2GIXSUAQASWCPU2C2WD5ZQPXX4JXQVYNCEBPB3TCONGPU6DXPVA, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.589671408 (536h28m25.460754454s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:25.596618467 (536h28m25.467701512s) [noise=true , address=45.91.116.7:4009 , piece_id=PM2PRTZVIIFRSURNFEJWFQH2F6IVSQUDVHI22TXBZ32X4CS4Y7NQ, seg_pos={0 0}, upload=470490]: Uploading piece... Dec 27 23:57:27.288823307 (536h28m27.159906352s) [upload=470490]: Next returned nil segment Dec 27 23:57:27.288839587 (536h28m27.159922633s) [upload=470490]: Waiting for error group managing segments... Dec 27 23:57:27.590060449 (536h28m27.461143484s) [noise=true , address=145.79.227.17:4972 , piece_id=SEQBN337U6YRTR4DGZIBKYOL465T5RPLX5SG32RUTACGS2PAOO7Q, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.590570340 (536h28m27.461653375s) [noise=true , address=145.79.227.45:4231 , piece_id=SAEUUUODIGOA4JMJSM34YQWELB3BK2DSXIX7O5D2OF23F2HMDGHQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.598426689 (536h28m27.469509724s) [noise=true , address=145.79.227.52:4445 , piece_id=B2GIXSUAQASWCPU2C2WD5ZQPXX4JXQVYNCEBPB3TCONGPU6DXPVA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.598906132 (536h28m27.469989168s) [noise=true , address=145.79.227.7:4295 , piece_id=ZDQ6IT3V7KH4QR67V5CJM2G5A4MTKHW3CPHD2OCYPK7I3Z6ECEZA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.600271612 (536h28m27.471354647s) [noise=true , address=145.79.227.9:4367 , piece_id=SUH2FLIOESG4G77L7RJHCROCDAH5VQS627HMQRGV337GI42NNPDQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.605502683 (536h28m27.476585728s) [noise=true , address=145.79.227.15:4873 , piece_id=2GDEG74PPW63ZZKPBG3KHSSEZP2EM7VM7ZXJUCI5LRXNUGTNLNTQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.811362122 (536h28m27.682445157s) [noise=true , address=45.91.116.12:4227 , piece_id=BE3XNZOPIZWQXF65HFS44UWVFGCXJLRNVLKIOVTA4X5E6L2IMEXQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.814354940 (536h28m27.685437975s) [noise=true , address=77.90.31.14:4661 , piece_id=FWPDKOP77HVUG5XLSEJEOAGPCD42P5QONM3BWHU747VG6DFCJHUA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.831121607 (536h28m27.702204652s) [noise=true , address=45.91.116.9:4261 , piece_id=D2GEEL7H3YBOIYAZVMN62XFWSH2FTTACN56EECHFISFTEGY5AIQQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.837465406 (536h28m27.708548442s) [noise=true , address=45.91.116.5:4297 , piece_id=YCX6OBBLRR3N4G7GY5NA3CISZRZ7M3QKOJ4M6LEYDTTDIBTCV6CQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.840602800 (536h28m27.711685846s) [noise=true , address=45.91.116.14:4191 , piece_id=YVLTDV3GZMCGGRZUJACGPAXAQT5BFJ5I6VYTL3DJO5QKV275VK5A, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.840961353 (536h28m27.712044398s) [noise=true , address=45.91.116.10:4083 , piece_id=27K2222F2VAVO3U7D6RK4HRSQ7V4UKFFQZRKM7MQ5F37UDHI7IHA, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.881586911 (536h28m27.752669946s) [noise=true , address=45.91.116.7:4009 , piece_id=PM2PRTZVIIFRSURNFEJWFQH2F6IVSQUDVHI22TXBZ32X4CS4Y7NQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: <nil> Dec 27 23:57:27.881631105 (536h28m27.75271414s) [seg_pos={0 0}, upload=470490]: Segment reached optimal threshold of 13 pieces. Dec 27 23:57:27.881919604 (536h28m27.753002659s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.881929853 (536h28m27.753012898s) [noise=true , address=145.79.227.35:4482 , piece_id=XM44PA26C5KWZUWOFSYAEGZFNQAVLGBEBYNDBNJXJHL7KK3X4C3Q, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882088035 (536h28m27.75317107s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.882108905 (536h28m27.75319195s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQMJGUEQU3SCIWECDNIQ32ITYRDRME47POJHX4S2XR55KI6SMDQ, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882337460 (536h28m27.753420505s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 23:57:27.882348260 (536h28m27.753431305s) [noise=true , address=145.79.227.33:4870 , piece_id=KLTY564KXIL6UWA54Q5GIBCKLBVMCA6CD7WZTSH5Z423DDF4W73A, seg_pos={0 0}, upload=470490]: Long tail cancellation due to optimal threshold Dec 27 23:57:27.882375522 (536h28m27.753458557s) [seg_pos={0 0}, upload=470490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:57:27.882466726 (536h28m27.753549771s) [upload=470490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:57:27.882473569 (536h28m27.753556604s) [upload=470490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:57:28.178184988 (536h28m28.049268023s) [upload=470490]: Done uploading segments... Dec 27 23:57:28.178203413 (536h28m28.049286448s) [upload=470490]: Upload finished. err: <nil> Duration: 30.002134645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.594907435s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:57:39 UTC) orbiter warning: context deadline exceeded Duration: 3.594933986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.......XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....] duration: 5.919766041s trace: https://jaeger.dev.storj.io/trace/7f8b1150e7eb0380 FAIL (2025-12-27 23:56:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 23:53:37.104859161 (536h24m36.975942196s) [upload=470436]: Got next segment. Inline: false Dec 27 23:53:37.104887284 (536h24m36.975970319s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:53:37.104891181 (536h24m36.975974226s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:53:37.351455197 (536h24m37.222538232s) [seg_pos={0 0}, upload=470436]: Begin upload segment... Dec 27 23:53:37.351498779 (536h24m37.222581825s) [seg_pos={0 0}, upload=470436]: Done begin upload segment. Dec 27 23:53:37.351522164 (536h24m37.222605199s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351540669 (536h24m37.222623714s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351924790 (536h24m37.223007836s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352213771 (536h24m37.223296816s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352515034 (536h24m37.223598079s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352824323 (536h24m37.223907359s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353135837 (536h24m37.224218872s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353437000 (536h24m37.224520036s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353865767 (536h24m37.224948802s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.354598612 (536h24m37.225681647s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355165592 (536h24m37.226248627s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355670453 (536h24m37.226753488s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356078521 (536h24m37.227161556s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356141451 (536h24m37.227224496s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356632736 (536h24m37.227715781s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.357139490 (536h24m37.228222535s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:39.067779458 (536h24m38.938862493s) [upload=470436]: Next returned nil segment Dec 27 23:53:39.067797722 (536h24m38.938880757s) [upload=470436]: Waiting for error group managing segments... Dec 27 23:53:39.353048703 (536h24m39.224131748s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.356722760 (536h24m39.227805795s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357213875 (536h24m39.22829691s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357773019 (536h24m39.228856064s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.358596047 (536h24m39.229679092s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370139186 (536h24m39.241222221s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370816557 (536h24m39.241899602s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.394854888 (536h24m39.265937923s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.592541824 (536h24m39.46362487s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.614842318 (536h24m39.485925363s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.621873657 (536h24m39.492956692s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.716604489 (536h24m39.587687534s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371352508 (536h24m40.242435543s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371373828 (536h24m40.242456863s) [seg_pos={0 0}, upload=470436]: Segment reached optimal threshold of 13 pieces. Dec 27 23:53:40.371700761 (536h24m40.242783806s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection Dec 27 23:53:40.371717192 (536h24m40.242800227s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371796793 (536h24m40.242879828s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection Dec 27 23:53:40.371805230 (536h24m40.242888265s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371870684 (536h24m40.242953719s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 23:53:40.371878800 (536h24m40.242961835s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371895952 (536h24m40.242978987s) [seg_pos={0 0}, upload=470436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:53:40.371970454 (536h24m40.243053489s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:53:40.371975754 (536h24m40.243058789s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:53:40.654891057 (536h24m40.525974102s) [upload=470436]: Done uploading segments... Dec 27 23:53:40.654918209 (536h24m40.526001244s) [upload=470436]: Upload finished. err: <nil> Duration: 5.919799254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.533073798s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:56:38 UTC) orbiter warning: context deadline exceeded Duration: 3.533094116s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.579271834s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:56:17 UTC) orbiter warning: context deadline exceeded Duration: 3.579288997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [........XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX...] duration: 6.936619191s trace: https://jaeger.dev.storj.io/trace/7f8b1150e7eb0380 FAIL (2025-12-27 23:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 23:53:37.104859161 (536h24m36.975942196s) [upload=470436]: Got next segment. Inline: false Dec 27 23:53:37.104887284 (536h24m36.975970319s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:53:37.104891181 (536h24m36.975974226s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:53:37.351455197 (536h24m37.222538232s) [seg_pos={0 0}, upload=470436]: Begin upload segment... Dec 27 23:53:37.351498779 (536h24m37.222581825s) [seg_pos={0 0}, upload=470436]: Done begin upload segment. Dec 27 23:53:37.351522164 (536h24m37.222605199s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351540669 (536h24m37.222623714s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351924790 (536h24m37.223007836s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352213771 (536h24m37.223296816s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352515034 (536h24m37.223598079s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352824323 (536h24m37.223907359s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353135837 (536h24m37.224218872s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353437000 (536h24m37.224520036s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353865767 (536h24m37.224948802s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.354598612 (536h24m37.225681647s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355165592 (536h24m37.226248627s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355670453 (536h24m37.226753488s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356078521 (536h24m37.227161556s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356141451 (536h24m37.227224496s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356632736 (536h24m37.227715781s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.357139490 (536h24m37.228222535s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:39.067779458 (536h24m38.938862493s) [upload=470436]: Next returned nil segment Dec 27 23:53:39.067797722 (536h24m38.938880757s) [upload=470436]: Waiting for error group managing segments... Dec 27 23:53:39.353048703 (536h24m39.224131748s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.356722760 (536h24m39.227805795s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357213875 (536h24m39.22829691s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357773019 (536h24m39.228856064s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.358596047 (536h24m39.229679092s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370139186 (536h24m39.241222221s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370816557 (536h24m39.241899602s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.394854888 (536h24m39.265937923s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.592541824 (536h24m39.46362487s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.614842318 (536h24m39.485925363s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.621873657 (536h24m39.492956692s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.716604489 (536h24m39.587687534s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371352508 (536h24m40.242435543s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371373828 (536h24m40.242456863s) [seg_pos={0 0}, upload=470436]: Segment reached optimal threshold of 13 pieces. Dec 27 23:53:40.371700761 (536h24m40.242783806s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection Dec 27 23:53:40.371717192 (536h24m40.242800227s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371796793 (536h24m40.242879828s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection Dec 27 23:53:40.371805230 (536h24m40.242888265s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371870684 (536h24m40.242953719s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 23:53:40.371878800 (536h24m40.242961835s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371895952 (536h24m40.242978987s) [seg_pos={0 0}, upload=470436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:53:40.371970454 (536h24m40.243053489s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:53:40.371975754 (536h24m40.243058789s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:53:40.654891057 (536h24m40.525974102s) [upload=470436]: Done uploading segments... Dec 27 23:53:40.654918209 (536h24m40.526001244s) [upload=470436]: Upload finished. err: <nil> Duration: 6.936645922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.368950591s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:55:35 UTC) orbiter warning: context deadline exceeded Duration: 3.369063044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.527634044s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:54:50 UTC) orbiter warning: context deadline exceeded Duration: 3.527687886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.........XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX..] duration: 6.982685234s trace: https://jaeger.dev.storj.io/trace/7f8b1150e7eb0380 FAIL (2025-12-27 23:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 23:53:37.104859161 (536h24m36.975942196s) [upload=470436]: Got next segment. Inline: false Dec 27 23:53:37.104887284 (536h24m36.975970319s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:53:37.104891181 (536h24m36.975974226s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:53:37.351455197 (536h24m37.222538232s) [seg_pos={0 0}, upload=470436]: Begin upload segment... Dec 27 23:53:37.351498779 (536h24m37.222581825s) [seg_pos={0 0}, upload=470436]: Done begin upload segment. Dec 27 23:53:37.351522164 (536h24m37.222605199s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351540669 (536h24m37.222623714s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351924790 (536h24m37.223007836s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352213771 (536h24m37.223296816s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352515034 (536h24m37.223598079s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352824323 (536h24m37.223907359s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353135837 (536h24m37.224218872s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353437000 (536h24m37.224520036s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353865767 (536h24m37.224948802s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.354598612 (536h24m37.225681647s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355165592 (536h24m37.226248627s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355670453 (536h24m37.226753488s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356078521 (536h24m37.227161556s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356141451 (536h24m37.227224496s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356632736 (536h24m37.227715781s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.357139490 (536h24m37.228222535s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:39.067779458 (536h24m38.938862493s) [upload=470436]: Next returned nil segment Dec 27 23:53:39.067797722 (536h24m38.938880757s) [upload=470436]: Waiting for error group managing segments... Dec 27 23:53:39.353048703 (536h24m39.224131748s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.356722760 (536h24m39.227805795s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357213875 (536h24m39.22829691s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357773019 (536h24m39.228856064s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.358596047 (536h24m39.229679092s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370139186 (536h24m39.241222221s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370816557 (536h24m39.241899602s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.394854888 (536h24m39.265937923s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.592541824 (536h24m39.46362487s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.614842318 (536h24m39.485925363s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.621873657 (536h24m39.492956692s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.716604489 (536h24m39.587687534s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371352508 (536h24m40.242435543s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371373828 (536h24m40.242456863s) [seg_pos={0 0}, upload=470436]: Segment reached optimal threshold of 13 pieces. Dec 27 23:53:40.371700761 (536h24m40.242783806s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection Dec 27 23:53:40.371717192 (536h24m40.242800227s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371796793 (536h24m40.242879828s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection Dec 27 23:53:40.371805230 (536h24m40.242888265s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371870684 (536h24m40.242953719s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 23:53:40.371878800 (536h24m40.242961835s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371895952 (536h24m40.242978987s) [seg_pos={0 0}, upload=470436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:53:40.371970454 (536h24m40.243053489s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:53:40.371975754 (536h24m40.243058789s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:53:40.654891057 (536h24m40.525974102s) [upload=470436]: Done uploading segments... Dec 27 23:53:40.654918209 (536h24m40.526001244s) [upload=470436]: Upload finished. err: <nil> Duration: 6.982713016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..........XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX.] duration: 7.980984472s trace: https://jaeger.dev.storj.io/trace/7f8b1150e7eb0380 FAIL (2025-12-27 23:54:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 23:53:37.104859161 (536h24m36.975942196s) [upload=470436]: Got next segment. Inline: false Dec 27 23:53:37.104887284 (536h24m36.975970319s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:53:37.104891181 (536h24m36.975974226s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:53:37.351455197 (536h24m37.222538232s) [seg_pos={0 0}, upload=470436]: Begin upload segment... Dec 27 23:53:37.351498779 (536h24m37.222581825s) [seg_pos={0 0}, upload=470436]: Done begin upload segment. Dec 27 23:53:37.351522164 (536h24m37.222605199s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351540669 (536h24m37.222623714s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351924790 (536h24m37.223007836s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352213771 (536h24m37.223296816s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352515034 (536h24m37.223598079s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352824323 (536h24m37.223907359s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353135837 (536h24m37.224218872s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353437000 (536h24m37.224520036s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353865767 (536h24m37.224948802s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.354598612 (536h24m37.225681647s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355165592 (536h24m37.226248627s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355670453 (536h24m37.226753488s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356078521 (536h24m37.227161556s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356141451 (536h24m37.227224496s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356632736 (536h24m37.227715781s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.357139490 (536h24m37.228222535s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:39.067779458 (536h24m38.938862493s) [upload=470436]: Next returned nil segment Dec 27 23:53:39.067797722 (536h24m38.938880757s) [upload=470436]: Waiting for error group managing segments... Dec 27 23:53:39.353048703 (536h24m39.224131748s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.356722760 (536h24m39.227805795s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357213875 (536h24m39.22829691s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357773019 (536h24m39.228856064s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.358596047 (536h24m39.229679092s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370139186 (536h24m39.241222221s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370816557 (536h24m39.241899602s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.394854888 (536h24m39.265937923s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.592541824 (536h24m39.46362487s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.614842318 (536h24m39.485925363s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.621873657 (536h24m39.492956692s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.716604489 (536h24m39.587687534s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371352508 (536h24m40.242435543s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371373828 (536h24m40.242456863s) [seg_pos={0 0}, upload=470436]: Segment reached optimal threshold of 13 pieces. Dec 27 23:53:40.371700761 (536h24m40.242783806s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection Dec 27 23:53:40.371717192 (536h24m40.242800227s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371796793 (536h24m40.242879828s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection Dec 27 23:53:40.371805230 (536h24m40.242888265s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371870684 (536h24m40.242953719s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 23:53:40.371878800 (536h24m40.242961835s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371895952 (536h24m40.242978987s) [seg_pos={0 0}, upload=470436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:53:40.371970454 (536h24m40.243053489s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:53:40.371975754 (536h24m40.243058789s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:53:40.654891057 (536h24m40.525974102s) [upload=470436]: Done uploading segments... Dec 27 23:53:40.654918209 (536h24m40.526001244s) [upload=470436]: Upload finished. err: <nil> Duration: 7.981005852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.49571504s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:54:12 UTC) orbiter warning: context deadline exceeded Duration: 3.496107286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...........XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX] duration: 30.002822158s trace: https://jaeger.dev.storj.io/trace/7f8b1150e7eb0380 FAIL (2025-12-27 23:54:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 23:53:37.104859161 (536h24m36.975942196s) [upload=470436]: Got next segment. Inline: false Dec 27 23:53:37.104887284 (536h24m36.975970319s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:53:37.104891181 (536h24m36.975974226s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:53:37.351455197 (536h24m37.222538232s) [seg_pos={0 0}, upload=470436]: Begin upload segment... Dec 27 23:53:37.351498779 (536h24m37.222581825s) [seg_pos={0 0}, upload=470436]: Done begin upload segment. Dec 27 23:53:37.351522164 (536h24m37.222605199s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351540669 (536h24m37.222623714s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.351924790 (536h24m37.223007836s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352213771 (536h24m37.223296816s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352515034 (536h24m37.223598079s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.352824323 (536h24m37.223907359s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353135837 (536h24m37.224218872s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353437000 (536h24m37.224520036s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.353865767 (536h24m37.224948802s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.354598612 (536h24m37.225681647s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355165592 (536h24m37.226248627s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.355670453 (536h24m37.226753488s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356078521 (536h24m37.227161556s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356141451 (536h24m37.227224496s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.356632736 (536h24m37.227715781s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:37.357139490 (536h24m37.228222535s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Uploading piece... Dec 27 23:53:39.067779458 (536h24m38.938862493s) [upload=470436]: Next returned nil segment Dec 27 23:53:39.067797722 (536h24m38.938880757s) [upload=470436]: Waiting for error group managing segments... Dec 27 23:53:39.353048703 (536h24m39.224131748s) [noise=true , address=145.79.227.16:4909 , piece_id=R7MGZ7ERIM443URINUDZVLKKOUDC4RBCWKKHQ27DIC5MLJRDTOAA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.356722760 (536h24m39.227805795s) [noise=true , address=145.79.227.45:4230 , piece_id=WS2UXVE3KL3PB5CAGGCECVE7ST5DMH2V5RF6KWB4RASCMUDFFQQQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357213875 (536h24m39.22829691s) [noise=true , address=145.79.227.11:4729 , piece_id=HUXK7SCZNJGBZIJAZL5FYREP76KWVNBKHS5SFDV27NHX7VYDABTQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.357773019 (536h24m39.228856064s) [noise=true , address=145.79.227.19:4077 , piece_id=WXN2IZS3TVHOH326OBATXGFAUDJDSABGMOMG65HGY5YXG435MT7Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.358596047 (536h24m39.229679092s) [noise=true , address=145.79.227.32:4836 , piece_id=YODT2FWF5ADURAMWFXREUNEQL3DNISBWYYM6WRUNOMAI5ICSKUBA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370139186 (536h24m39.241222221s) [noise=true , address=145.79.227.48:4303 , piece_id=QYYTE5OA552KK2BXDY4IGJKY6UOKU2RN7YKWMJK54FFL4DX6JWQA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.370816557 (536h24m39.241899602s) [noise=true , address=145.79.227.39:4015 , piece_id=EF4G3AZE5MLIYLCZVQZUIZOPVIX3442SAKZYJ4YLOFBPAFAEB25A, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.394854888 (536h24m39.265937923s) [noise=true , address=145.79.227.14:4839 , piece_id=EZJQCDMPEVS7VDJMWRUPCBEJUPMLKBJQZCM6PLRE6JLVJSWYYCRA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.592541824 (536h24m39.46362487s) [noise=true , address=77.90.31.21:4880 , piece_id=ZQRNTR4A2TXPDVP3KTU4A64CGXMQOBWCSUNQZNI4HABW4ZAIJQPA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.614842318 (536h24m39.485925363s) [noise=true , address=45.91.116.10:4083 , piece_id=MGJIMVVN5BBFIOXKCUHS5XRIBLLSQCIVFJ247IKWZIHCBUBN4T3Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.621873657 (536h24m39.492956692s) [noise=true , address=45.91.116.9:4264 , piece_id=YLHSSGGML4O6INM2OUMLH3V7YA4FERANVEDJNRHHZV5FDYGR5GXA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:39.716604489 (536h24m39.587687534s) [noise=true , address=45.91.116.11:4120 , piece_id=4FNQXBSNUISTNOKVVLWA337CKQZEE575S44KXZKV62W27AM6LTLA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371352508 (536h24m40.242435543s) [noise=true , address=45.91.116.8:4045 , piece_id=7APZAJHGTVQGSN3YZWGI4P4CLISA3K2JF75VVAVZBCSSE6XXP2NQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: <nil> Dec 27 23:53:40.371373828 (536h24m40.242456863s) [seg_pos={0 0}, upload=470436]: Segment reached optimal threshold of 13 pieces. Dec 27 23:53:40.371700761 (536h24m40.242783806s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.41:4087: use of closed network connection Dec 27 23:53:40.371717192 (536h24m40.242800227s) [noise=true , address=145.79.227.41:4087 , piece_id=ZRFJRV7437TKYWKADJBQ3KBW4P65KGGXZVU4YL6KBHHYL7SPNMDA, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371796793 (536h24m40.242879828s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47600->145.79.227.51:4409: use of closed network connection Dec 27 23:53:40.371805230 (536h24m40.242888265s) [noise=true , address=145.79.227.51:4409 , piece_id=66A25DCPJLDHZ463RKPDN5RVR7TLZ7DTZMSWQ3XQ3VNRXCQ65GUQ, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371870684 (536h24m40.242953719s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 23:53:40.371878800 (536h24m40.242961835s) [noise=true , address=77.90.31.13:4627 , piece_id=ES5EYJIB2AQ2C7VMLZJQ52E433IT7EL4RNHPZJSQY3MNKGKJ2D2Q, seg_pos={0 0}, upload=470436]: Long tail cancellation due to optimal threshold Dec 27 23:53:40.371895952 (536h24m40.242978987s) [seg_pos={0 0}, upload=470436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:53:40.371970454 (536h24m40.243053489s) [upload=470436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:53:40.371975754 (536h24m40.243058789s) [upload=470436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:53:40.654891057 (536h24m40.525974102s) [upload=470436]: Done uploading segments... Dec 27 23:53:40.654918209 (536h24m40.526001244s) [upload=470436]: Upload finished. err: <nil> Duration: 30.002866893s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.538178908s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:53:23 UTC) orbiter warning: context deadline exceeded Duration: 3.53821689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [............XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XX] duration: 30.003084748s trace: https://jaeger.dev.storj.io/trace/6132d70b2afd1331 FAIL (2025-12-27 23:53:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: GW4SC4IEYGTM3GUQEABYPUADGCC2DZXMLHLFNWP2V4NMR5N54G2A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: ABVR2FI2KTFOMFMGNPZVZZLOWJN3JVKIL7HLLVJHCX3LPAREIIDQ) context deadline exceeded ======LOGS====== 23:52:44.905150791 (536h23m44.776233836s) [upload=470424]: Starting upload Dec 27 23:52:44.905241814 (536h23m44.776324859s) [upload=470424]: Done starting upload Dec 27 23:52:44.905290407 (536h23m44.776373442s) [upload=470424]: Uploading segments... Dec 27 23:52:44.905509665 (536h23m44.7765927s) [upload=470424]: Got next segment. Inline: false Dec 27 23:52:44.905541344 (536h23m44.776624389s) [upload=470424]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:52:44.905546785 (536h23m44.77662982s) [upload=470424]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:52:45.155159454 (536h23m45.026242489s) [seg_pos={0 0}, upload=470424]: Begin upload segment... Dec 27 23:52:45.155201636 (536h23m45.026284671s) [seg_pos={0 0}, upload=470424]: Done begin upload segment. Dec 27 23:52:45.155229327 (536h23m45.026312363s) [noise=true , address=145.79.227.9:4365 , piece_id=TI5OYJ7EWQXIHB4R7U6LJTCTJ7NETLBNN3CNWO65OPBU7RIBMHPA, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.155259916 (536h23m45.026342951s) [noise=true , address=77.90.31.17:4773 , piece_id=ABVR2FI2KTFOMFMGNPZVZZLOWJN3JVKIL7HLLVJHCX3LPAREIIDQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.155786979 (536h23m45.026870014s) [noise=true , address=45.91.116.6:4976 , piece_id=WEVBXMC3YJAWDRKPCT6GUVB4FRKPB32ECC7MYRPYELBEFM7GA7MA, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.156198423 (536h23m45.027281468s) [noise=true , address=145.79.227.11:4730 , piece_id=LDP7EDDISB5CDDYVG64H7E7WBG6F5SQ5OKDPWN5MQSRRD3J45WVQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.156477324 (536h23m45.027560359s) [noise=true , address=77.90.31.21:4877 , piece_id=SVSUW7IEKGZ6MRIXL64MUGDUXJXRPBCSLPCTTRBZGQMJSLXXQWZA, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.156755113 (536h23m45.027838148s) [noise=true , address=77.90.31.16:4735 , piece_id=CDMSSBQVDUG2KRODCAD7WN3RDQGMBE53XSNJIIZ3MM3ADKMVD3ZQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.156849012 (536h23m45.027932057s) [noise=true , address=77.90.31.14:4663 , piece_id=GW4SC4IEYGTM3GUQEABYPUADGCC2DZXMLHLFNWP2V4NMR5N54G2A, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157128454 (536h23m45.028211489s) [noise=true , address=45.91.116.14:4190 , piece_id=I7G37I7ITAVRDYXUOZIZ5EOQX7I6JYDKMLT2YNXLKWZSXXD4DM4Q, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157236059 (536h23m45.028319094s) [noise=true , address=145.79.227.7:4295 , piece_id=NVWE5JEV6N42LOOERPQJKOAESKZG7M7OQE2P5T6MHH4BJRQFNDGA, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157411753 (536h23m45.028494799s) [noise=true , address=45.91.116.11:4118 , piece_id=FIHHF7ZBCPWWVVMBPHGR35MMSF2C5OTHFIOFKFRTCY7JKN3KUFPQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157558282 (536h23m45.028641317s) [noise=true , address=145.79.227.41:4086 , piece_id=ZK3WD7DQVH6GEPAAZOO5U6EC2O5LRDLXORKUAGNRE4HQUU46AF4Q, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157659655 (536h23m45.02874269s) [noise=true , address=145.79.227.48:4303 , piece_id=SF7LQQSOMI54DMTLHFIOWTEBEZBIC34XI5L3WHYQO23J2ARQIKPQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157877691 (536h23m45.028960726s) [noise=true , address=145.79.227.50:4374 , piece_id=IQGRWJ4TBJK75OWOSBCRMJQ6KBQXSIU3WL4VYLF7JT4V5M2ORS3Q, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.157952814 (536h23m45.029035849s) [noise=true , address=145.79.227.18:4007 , piece_id=PQ72XQNNDJQMUHR3QBDCJGPX3PBCHNBADNRZZQJTTPAX4YH56APQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.158200486 (536h23m45.029283521s) [noise=true , address=145.79.227.45:4229 , piece_id=SSL3UAKNFF7GMH76MPJOD2L45K4EIHGSU4ITJMRTATKTZVISQNAQ, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:45.158588544 (536h23m45.02967158s) [noise=true , address=145.79.227.32:4836 , piece_id=UHTRCKMMCM3JKR5JBIX3TKONDXNYBHV3XPY77UEYOYIB5IXXQLZA, seg_pos={0 0}, upload=470424]: Uploading piece... Dec 27 23:52:46.875803950 (536h23m46.746886985s) [upload=470424]: Next returned nil segment Dec 27 23:52:46.875816674 (536h23m46.746899709s) [upload=470424]: Waiting for error group managing segments... Dec 27 23:52:47.156297321 (536h23m47.027380356s) [noise=true , address=145.79.227.11:4730 , piece_id=LDP7EDDISB5CDDYVG64H7E7WBG6F5SQ5OKDPWN5MQSRRD3J45WVQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.159580653 (536h23m47.030663688s) [noise=true , address=145.79.227.45:4229 , piece_id=SSL3UAKNFF7GMH76MPJOD2L45K4EIHGSU4ITJMRTATKTZVISQNAQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.170834302 (536h23m47.041917337s) [noise=true , address=145.79.227.7:4295 , piece_id=NVWE5JEV6N42LOOERPQJKOAESKZG7M7OQE2P5T6MHH4BJRQFNDGA, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.172101936 (536h23m47.043184971s) [noise=true , address=145.79.227.9:4365 , piece_id=TI5OYJ7EWQXIHB4R7U6LJTCTJ7NETLBNN3CNWO65OPBU7RIBMHPA, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.178674400 (536h23m47.049757475s) [noise=true , address=145.79.227.50:4374 , piece_id=IQGRWJ4TBJK75OWOSBCRMJQ6KBQXSIU3WL4VYLF7JT4V5M2ORS3Q, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.179480586 (536h23m47.050563631s) [noise=true , address=145.79.227.41:4086 , piece_id=ZK3WD7DQVH6GEPAAZOO5U6EC2O5LRDLXORKUAGNRE4HQUU46AF4Q, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.180077382 (536h23m47.051160418s) [noise=true , address=145.79.227.48:4303 , piece_id=SF7LQQSOMI54DMTLHFIOWTEBEZBIC34XI5L3WHYQO23J2ARQIKPQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.184586037 (536h23m47.055669082s) [noise=true , address=145.79.227.32:4836 , piece_id=UHTRCKMMCM3JKR5JBIX3TKONDXNYBHV3XPY77UEYOYIB5IXXQLZA, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.335795292 (536h23m47.206878327s) [noise=true , address=77.90.31.17:4773 , piece_id=ABVR2FI2KTFOMFMGNPZVZZLOWJN3JVKIL7HLLVJHCX3LPAREIIDQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.360334207 (536h23m47.231417252s) [noise=true , address=77.90.31.16:4735 , piece_id=CDMSSBQVDUG2KRODCAD7WN3RDQGMBE53XSNJIIZ3MM3ADKMVD3ZQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.443180379 (536h23m47.314263414s) [noise=true , address=45.91.116.11:4118 , piece_id=FIHHF7ZBCPWWVVMBPHGR35MMSF2C5OTHFIOFKFRTCY7JKN3KUFPQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.443813795 (536h23m47.31489684s) [noise=true , address=77.90.31.14:4663 , piece_id=GW4SC4IEYGTM3GUQEABYPUADGCC2DZXMLHLFNWP2V4NMR5N54G2A, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.447556943 (536h23m47.318639978s) [noise=true , address=45.91.116.6:4976 , piece_id=WEVBXMC3YJAWDRKPCT6GUVB4FRKPB32ECC7MYRPYELBEFM7GA7MA, seg_pos={0 0}, upload=470424]: Done uploading piece. err: <nil> Dec 27 23:52:47.447587511 (536h23m47.318670546s) [seg_pos={0 0}, upload=470424]: Segment reached optimal threshold of 13 pieces. Dec 27 23:52:47.447764498 (536h23m47.318847533s) [noise=true , address=45.91.116.14:4190 , piece_id=I7G37I7ITAVRDYXUOZIZ5EOQX7I6JYDKMLT2YNXLKWZSXXD4DM4Q, seg_pos={0 0}, upload=470424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 23:52:47.447770449 (536h23m47.318853494s) [noise=true , address=45.91.116.14:4190 , piece_id=I7G37I7ITAVRDYXUOZIZ5EOQX7I6JYDKMLT2YNXLKWZSXXD4DM4Q, seg_pos={0 0}, upload=470424]: Long tail cancellation due to optimal threshold Dec 27 23:52:47.447889145 (536h23m47.31897218s) [noise=true , address=77.90.31.21:4877 , piece_id=SVSUW7IEKGZ6MRIXL64MUGDUXJXRPBCSLPCTTRBZGQMJSLXXQWZA, seg_pos={0 0}, upload=470424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 23:52:47.447893994 (536h23m47.318977029s) [noise=true , address=77.90.31.21:4877 , piece_id=SVSUW7IEKGZ6MRIXL64MUGDUXJXRPBCSLPCTTRBZGQMJSLXXQWZA, seg_pos={0 0}, upload=470424]: Long tail cancellation due to optimal threshold Dec 27 23:52:47.448054159 (536h23m47.319137194s) [noise=true , address=145.79.227.18:4007 , piece_id=PQ72XQNNDJQMUHR3QBDCJGPX3PBCHNBADNRZZQJTTPAX4YH56APQ, seg_pos={0 0}, upload=470424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:49634->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:49634->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49634->145.79.227.18:4007: use of closed network connection Dec 27 23:52:47.448065901 (536h23m47.319148947s) [noise=true , address=145.79.227.18:4007 , piece_id=PQ72XQNNDJQMUHR3QBDCJGPX3PBCHNBADNRZZQJTTPAX4YH56APQ, seg_pos={0 0}, upload=470424]: Long tail cancellation due to optimal threshold Dec 27 23:52:47.448090158 (536h23m47.319173203s) [seg_pos={0 0}, upload=470424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:52:47.448174027 (536h23m47.319257062s) [upload=470424]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:52:47.448179228 (536h23m47.319262263s) [upload=470424]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:52:47.739847028 (536h23m47.610930063s) [upload=470424]: Done uploading segments... Dec 27 23:52:47.739869842 (536h23m47.610952877s) [upload=470424]: Upload finished. err: <nil> Duration: 30.003179027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.............XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.X] duration: 30.002832913s trace: https://jaeger.dev.storj.io/trace/106048f6da965ee8 FAIL (2025-12-27 23:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: EOM5LRNJPY6GV2W7KXTZRF2TJJKRWYESWSTJO2QFMMUMYJPRS6ZA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: KAK3QL4OGPCUC5UUDGDVQX3GXF7YF4CMNWFDOMYEOUREISYIS2AA) context deadline exceeded ======LOGS====== Dec 27 23:52:04.126948235 (536h23m3.99803128s) [upload=470414]: Starting upload Dec 27 23:52:04.127025462 (536h23m3.998108497s) [upload=470414]: Done starting upload Dec 27 23:52:04.127252113 (536h23m3.998335159s) [upload=470414]: Uploading segments... Dec 27 23:52:04.127344580 (536h23m3.998427655s) [upload=470414]: Got next segment. Inline: false Dec 27 23:52:04.127432196 (536h23m3.998515241s) [upload=470414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:52:04.127443919 (536h23m3.998526994s) [upload=470414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:52:04.370824826 (536h23m4.241907861s) [seg_pos={0 0}, upload=470414]: Begin upload segment... Dec 27 23:52:04.370878788 (536h23m4.241961823s) [seg_pos={0 0}, upload=470414]: Done begin upload segment. Dec 27 23:52:04.370928784 (536h23m4.242011829s) [noise=true , address=45.91.116.9:4264 , piece_id=7QXMYAEZCYCHLVR5SLRID46HSP5F576TFKEFYYMRLJBE2AKKX27Q, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.370954503 (536h23m4.242037548s) [noise=true , address=145.79.227.48:4302 , piece_id=7TQXAPGQRO7EWA4IQ5LSGAOCA7JOYISBHPZFRZW6X5NKK6ZISK6A, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.371054733 (536h23m4.242137778s) [noise=true , address=45.91.116.12:4225 , piece_id=3BZE2QFVBR2RU5JJIB6FXZ36R5JPJW7BRO5JO24253JCTGFT7L6Q, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.371695624 (536h23m4.242778669s) [noise=true , address=77.90.31.21:4878 , piece_id=T35DMON7ZHVOH4JYEGPCG6PWMYBYMEUV2AXGWTJLL3DTCMQJVGDQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.372140582 (536h23m4.243223627s) [noise=true , address=145.79.227.9:4365 , piece_id=ZYZMX6NN3MQOWV6KULB4VOIZ7VSLBEGWH7TTTRY5G6V2N2LPXUZQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.372575429 (536h23m4.243658464s) [noise=true , address=45.91.116.7:4010 , piece_id=EOM5LRNJPY6GV2W7KXTZRF2TJJKRWYESWSTJO2QFMMUMYJPRS6ZA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.372584777 (536h23m4.243667822s) [noise=true , address=145.79.227.41:4085 , piece_id=SMYGQEWFB5DVG4M3HGLNW46RJPVC3EYUPGQFEQHMVMELAAZYG5AQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.372940014 (536h23m4.244023059s) [noise=true , address=145.79.227.38:4977 , piece_id=EIMK7KU3P5HZRKCTNWLTEHNRQRGZS33UWB5A74FDMFRV2MJTHFYA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.373020487 (536h23m4.244103542s) [noise=true , address=145.79.227.37:4567 , piece_id=MFZLIQEUTZYLVXD3W4AC5XJ7OFLK2KD5L4DTFWLJSIT35S57IPKA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.373279339 (536h23m4.244362374s) [noise=true , address=77.90.31.14:4661 , piece_id=CBFWQYJSRLMU7PVGAXYRTR4JVXDC66OGVCJORZC5RTYWHCIUMBPQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.373600612 (536h23m4.244683647s) [noise=true , address=145.79.227.31:4798 , piece_id=VVVVKELAAGGVNQOAF4WKE5R3WUKPLMFAUE5X44CLCRCU4QKVVXOA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.374007255 (536h23m4.24509029s) [noise=true , address=77.90.31.18:4808 , piece_id=OJXW7KZYM5NWYCNHEBQBR2DLC3WOJTONY2OI7MFQSRSNG3OSHOMQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.374454838 (536h23m4.245537873s) [noise=true , address=145.79.227.4:4186 , piece_id=TXTCHYCSMEJYGFZJSL4YWMVST7TT2OBFAYQJGDBWQNEWYH2GKGNQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.374861533 (536h23m4.245944578s) [noise=true , address=45.91.116.6:4976 , piece_id=KAK3QL4OGPCUC5UUDGDVQX3GXF7YF4CMNWFDOMYEOUREISYIS2AA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.375247237 (536h23m4.246330282s) [noise=true , address=145.79.227.51:4409 , piece_id=PPHCE447FSGS7H7OHULDCMDPVJM2VYD5T7FJEEYS3KD7WFUZEUIA, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:04.377429613 (536h23m4.248512658s) [noise=true , address=145.79.227.45:4229 , piece_id=O4E6TPAEZ7OP4FQS6NIIR622QDHRJJPO4UA4LROW75VVJWZI4QZQ, seg_pos={0 0}, upload=470414]: Uploading piece... Dec 27 23:52:06.080273214 (536h23m5.951356249s) [upload=470414]: Next returned nil segment Dec 27 23:52:06.080310475 (536h23m5.95139353s) [upload=470414]: Waiting for error group managing segments... Dec 27 23:52:06.363609819 (536h23m6.234692854s) [noise=true , address=145.79.227.9:4365 , piece_id=ZYZMX6NN3MQOWV6KULB4VOIZ7VSLBEGWH7TTTRY5G6V2N2LPXUZQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.380138002 (536h23m6.251221037s) [noise=true , address=145.79.227.38:4977 , piece_id=EIMK7KU3P5HZRKCTNWLTEHNRQRGZS33UWB5A74FDMFRV2MJTHFYA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.383755280 (536h23m6.254838325s) [noise=true , address=145.79.227.45:4229 , piece_id=O4E6TPAEZ7OP4FQS6NIIR622QDHRJJPO4UA4LROW75VVJWZI4QZQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.387431680 (536h23m6.258514715s) [noise=true , address=145.79.227.31:4798 , piece_id=VVVVKELAAGGVNQOAF4WKE5R3WUKPLMFAUE5X44CLCRCU4QKVVXOA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.391434141 (536h23m6.262517186s) [noise=true , address=145.79.227.51:4409 , piece_id=PPHCE447FSGS7H7OHULDCMDPVJM2VYD5T7FJEEYS3KD7WFUZEUIA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.394780644 (536h23m6.265863689s) [noise=true , address=145.79.227.4:4186 , piece_id=TXTCHYCSMEJYGFZJSL4YWMVST7TT2OBFAYQJGDBWQNEWYH2GKGNQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.399003114 (536h23m6.270086149s) [noise=true , address=145.79.227.41:4085 , piece_id=SMYGQEWFB5DVG4M3HGLNW46RJPVC3EYUPGQFEQHMVMELAAZYG5AQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.573116772 (536h23m6.444199817s) [noise=true , address=77.90.31.18:4808 , piece_id=OJXW7KZYM5NWYCNHEBQBR2DLC3WOJTONY2OI7MFQSRSNG3OSHOMQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.616347671 (536h23m6.487430716s) [noise=true , address=77.90.31.21:4878 , piece_id=T35DMON7ZHVOH4JYEGPCG6PWMYBYMEUV2AXGWTJLL3DTCMQJVGDQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.618001180 (536h23m6.489084215s) [noise=true , address=77.90.31.14:4661 , piece_id=CBFWQYJSRLMU7PVGAXYRTR4JVXDC66OGVCJORZC5RTYWHCIUMBPQ, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.620724926 (536h23m6.491807971s) [noise=true , address=45.91.116.7:4010 , piece_id=EOM5LRNJPY6GV2W7KXTZRF2TJJKRWYESWSTJO2QFMMUMYJPRS6ZA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:06.693551057 (536h23m6.564634092s) [noise=true , address=45.91.116.9:4264 , piece_id=7QXMYAEZCYCHLVR5SLRID46HSP5F576TFKEFYYMRLJBE2AKKX27Q, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:07.323247986 (536h23m7.194331021s) [noise=true , address=45.91.116.6:4976 , piece_id=KAK3QL4OGPCUC5UUDGDVQX3GXF7YF4CMNWFDOMYEOUREISYIS2AA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: <nil> Dec 27 23:52:07.323268966 (536h23m7.194352001s) [seg_pos={0 0}, upload=470414]: Segment reached optimal threshold of 13 pieces. Dec 27 23:52:07.323468506 (536h23m7.194551541s) [noise=true , address=145.79.227.48:4302 , piece_id=7TQXAPGQRO7EWA4IQ5LSGAOCA7JOYISBHPZFRZW6X5NKK6ZISK6A, seg_pos={0 0}, upload=470414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 27 23:52:07.323476141 (536h23m7.194559186s) [noise=true , address=145.79.227.48:4302 , piece_id=7TQXAPGQRO7EWA4IQ5LSGAOCA7JOYISBHPZFRZW6X5NKK6ZISK6A, seg_pos={0 0}, upload=470414]: Long tail cancellation due to optimal threshold Dec 27 23:52:07.323576532 (536h23m7.194659577s) [noise=true , address=145.79.227.37:4567 , piece_id=MFZLIQEUTZYLVXD3W4AC5XJ7OFLK2KD5L4DTFWLJSIT35S57IPKA, seg_pos={0 0}, upload=470414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Dec 27 23:52:07.323592352 (536h23m7.194675397s) [noise=true , address=145.79.227.37:4567 , piece_id=MFZLIQEUTZYLVXD3W4AC5XJ7OFLK2KD5L4DTFWLJSIT35S57IPKA, seg_pos={0 0}, upload=470414]: Long tail cancellation due to optimal threshold Dec 27 23:52:07.323723602 (536h23m7.194806637s) [noise=true , address=45.91.116.12:4225 , piece_id=3BZE2QFVBR2RU5JJIB6FXZ36R5JPJW7BRO5JO24253JCTGFT7L6Q, seg_pos={0 0}, upload=470414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 23:52:07.323730014 (536h23m7.194813049s) [noise=true , address=45.91.116.12:4225 , piece_id=3BZE2QFVBR2RU5JJIB6FXZ36R5JPJW7BRO5JO24253JCTGFT7L6Q, seg_pos={0 0}, upload=470414]: Long tail cancellation due to optimal threshold Dec 27 23:52:07.323746396 (536h23m7.194829441s) [seg_pos={0 0}, upload=470414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:52:07.323812060 (536h23m7.194895095s) [upload=470414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:52:07.323817451 (536h23m7.194900496s) [upload=470414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:52:07.612802581 (536h23m7.483885616s) [upload=470414]: Done uploading segments... Dec 27 23:52:07.612822459 (536h23m7.483905504s) [upload=470414]: Upload finished. err: <nil> Duration: 30.002857609s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.374143856s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:52:23 UTC) orbiter warning: context deadline exceeded Duration: 3.374174044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.501622138s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:52:01 UTC) orbiter warning: context deadline exceeded Duration: 3.501649992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..............XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.] duration: 15.473576163s trace: https://jaeger.dev.storj.io/trace/d83b2995adec594 FAIL (2025-12-27 23:51:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A) context deadline exceeded ======LOGS====== s... Dec 27 23:49:53.792584821 (536h20m53.663667866s) [upload=470384]: Got next segment. Inline: false Dec 27 23:49:53.792621561 (536h20m53.663704606s) [upload=470384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:49:53.792626370 (536h20m53.663709405s) [upload=470384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:49:54.032551833 (536h20m53.903634888s) [seg_pos={0 0}, upload=470384]: Begin upload segment... Dec 27 23:49:54.032600166 (536h20m53.903683211s) [seg_pos={0 0}, upload=470384]: Done begin upload segment. Dec 27 23:49:54.032627818 (536h20m53.903710863s) [noise=true , address=145.79.227.5:4221 , piece_id=V6BJWTM5DDNW6GQGAH2A5LSP66V3KU4PBDQRWJKZ4Q7IHTSPRLTA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.032643107 (536h20m53.903726152s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033132379 (536h20m53.904215414s) [noise=true , address=45.91.116.8:4047 , piece_id=YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033522642 (536h20m53.904605687s) [noise=true , address=45.91.116.13:4155 , piece_id=E2PH5R64L7SJKFO6MIEOIQBY6RUQRZ7JWN2ZDMSJ4OPFWXVYW3HA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033960346 (536h20m53.905043391s) [noise=true , address=145.79.227.49:4337 , piece_id=BTOEY4JTYXYSHHPWNE4F3EFC222TP7CV5DETYXESMN3IHFWACA7A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.034472691 (536h20m53.905555736s) [noise=true , address=145.79.227.25:4618 , piece_id=LEKOWTLBNHVNH26VLGWUHPBQEOMRXVN3PPM4MPPKELYQLFCARF7Q, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.034894584 (536h20m53.905977629s) [noise=true , address=145.79.227.37:4567 , piece_id=BBCSD527H5AMFVJ3CIGEWETSAPHIUXATPXO7EUXE4TNMXFEAJLPQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035120204 (536h20m53.906203249s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035320546 (536h20m53.906403591s) [noise=true , address=145.79.227.48:4302 , piece_id=LXGVHZHN6N5G6HN57PDS5NCQYNVDQ66FOT2VBZUM5TWJMHY7AC6A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035736688 (536h20m53.906819733s) [noise=true , address=45.91.116.6:4975 , piece_id=PPCMP3EZKPSR3MKG5GLU6DBFNIIJOZ7QTNNKT6NFWITKTXJ7SQOQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035750544 (536h20m53.906833579s) [noise=true , address=145.79.227.11:4730 , piece_id=ZKPTQGIZZAI3T42ZFZYKPXHGOFJEMA2LDUO5IA5ISYD4PT6ELTNQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.036308316 (536h20m53.907391361s) [noise=true , address=45.91.116.9:4264 , piece_id=YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.036941011 (536h20m53.908024066s) [noise=true , address=45.91.116.7:4009 , piece_id=3ZRK2J6F7NHLNF6FELNSF6RRXHECLH6GCWOBQVUX3O5JMDOX64KA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.037561823 (536h20m53.908644868s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.038148420 (536h20m53.909231465s) [noise=true , address=145.79.227.36:4518 , piece_id=BUE6DDX7IXJVZ3FWLEIMINXS5BOJSBQWQ64LLTVRJDHL6MIHOPUA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.038187164 (536h20m53.909270199s) [noise=true , address=145.79.227.12:4767 , piece_id=D55M4EN5H5M3B2YHGDD6HKLVM3RDMTQUTCWWRVRL5ZGAWNR2B6JQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:55.748361325 (536h20m55.61944436s) [upload=470384]: Next returned nil segment Dec 27 23:49:55.748375892 (536h20m55.619458937s) [upload=470384]: Waiting for error group managing segments... Dec 27 23:49:56.028135006 (536h20m55.899218051s) [noise=true , address=145.79.227.36:4518 , piece_id=BUE6DDX7IXJVZ3FWLEIMINXS5BOJSBQWQ64LLTVRJDHL6MIHOPUA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.039866044 (536h20m55.910949089s) [noise=true , address=145.79.227.11:4730 , piece_id=ZKPTQGIZZAI3T42ZFZYKPXHGOFJEMA2LDUO5IA5ISYD4PT6ELTNQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.046130612 (536h20m55.917213647s) [noise=true , address=145.79.227.37:4567 , piece_id=BBCSD527H5AMFVJ3CIGEWETSAPHIUXATPXO7EUXE4TNMXFEAJLPQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.047418615 (536h20m55.91850166s) [noise=true , address=145.79.227.5:4221 , piece_id=V6BJWTM5DDNW6GQGAH2A5LSP66V3KU4PBDQRWJKZ4Q7IHTSPRLTA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.053789636 (536h20m55.924872671s) [noise=true , address=145.79.227.25:4618 , piece_id=LEKOWTLBNHVNH26VLGWUHPBQEOMRXVN3PPM4MPPKELYQLFCARF7Q, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.059014726 (536h20m55.930097761s) [noise=true , address=145.79.227.48:4302 , piece_id=LXGVHZHN6N5G6HN57PDS5NCQYNVDQ66FOT2VBZUM5TWJMHY7AC6A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.065164977 (536h20m55.936248012s) [noise=true , address=145.79.227.12:4767 , piece_id=D55M4EN5H5M3B2YHGDD6HKLVM3RDMTQUTCWWRVRL5ZGAWNR2B6JQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.147749461 (536h20m56.018832496s) [noise=true , address=145.79.227.49:4337 , piece_id=BTOEY4JTYXYSHHPWNE4F3EFC222TP7CV5DETYXESMN3IHFWACA7A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.261321269 (536h20m56.132404314s) [noise=true , address=45.91.116.13:4155 , piece_id=E2PH5R64L7SJKFO6MIEOIQBY6RUQRZ7JWN2ZDMSJ4OPFWXVYW3HA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.320094221 (536h20m56.191177256s) [noise=true , address=45.91.116.9:4264 , piece_id=YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.320473874 (536h20m56.191556909s) [noise=true , address=45.91.116.6:4975 , piece_id=PPCMP3EZKPSR3MKG5GLU6DBFNIIJOZ7QTNNKT6NFWITKTXJ7SQOQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.343378848 (536h20m56.214461883s) [noise=true , address=45.91.116.7:4009 , piece_id=3ZRK2J6F7NHLNF6FELNSF6RRXHECLH6GCWOBQVUX3O5JMDOX64KA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.972167347 (536h20m56.843250382s) [noise=true , address=45.91.116.8:4047 , piece_id=YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.972188226 (536h20m56.843271261s) [seg_pos={0 0}, upload=470384]: Segment reached optimal threshold of 13 pieces. Dec 27 23:49:56.972491374 (536h20m56.843574409s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 23:49:56.972500511 (536h20m56.843583556s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972537631 (536h20m56.843620676s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection Dec 27 23:49:56.972554634 (536h20m56.843637679s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972826402 (536h20m56.843909437s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection Dec 27 23:49:56.972835820 (536h20m56.843918875s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972856730 (536h20m56.843939775s) [seg_pos={0 0}, upload=470384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:49:56.972952071 (536h20m56.844035106s) [upload=470384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:49:56.972959264 (536h20m56.844042299s) [upload=470384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:49:57.260037595 (536h20m57.13112065s) [upload=470384]: Done uploading segments... Dec 27 23:49:57.260078192 (536h20m57.131161237s) [upload=470384]: Upload finished. err: <nil> Duration: 15.473604146s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.567152365s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:50:49 UTC) orbiter warning: context deadline exceeded Duration: 3.567184456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...............XXX.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX] duration: 30.002962033s trace: https://jaeger.dev.storj.io/trace/d83b2995adec594 FAIL (2025-12-27 23:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A) context deadline exceeded ======LOGS====== s... Dec 27 23:49:53.792584821 (536h20m53.663667866s) [upload=470384]: Got next segment. Inline: false Dec 27 23:49:53.792621561 (536h20m53.663704606s) [upload=470384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:49:53.792626370 (536h20m53.663709405s) [upload=470384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:49:54.032551833 (536h20m53.903634888s) [seg_pos={0 0}, upload=470384]: Begin upload segment... Dec 27 23:49:54.032600166 (536h20m53.903683211s) [seg_pos={0 0}, upload=470384]: Done begin upload segment. Dec 27 23:49:54.032627818 (536h20m53.903710863s) [noise=true , address=145.79.227.5:4221 , piece_id=V6BJWTM5DDNW6GQGAH2A5LSP66V3KU4PBDQRWJKZ4Q7IHTSPRLTA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.032643107 (536h20m53.903726152s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033132379 (536h20m53.904215414s) [noise=true , address=45.91.116.8:4047 , piece_id=YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033522642 (536h20m53.904605687s) [noise=true , address=45.91.116.13:4155 , piece_id=E2PH5R64L7SJKFO6MIEOIQBY6RUQRZ7JWN2ZDMSJ4OPFWXVYW3HA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.033960346 (536h20m53.905043391s) [noise=true , address=145.79.227.49:4337 , piece_id=BTOEY4JTYXYSHHPWNE4F3EFC222TP7CV5DETYXESMN3IHFWACA7A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.034472691 (536h20m53.905555736s) [noise=true , address=145.79.227.25:4618 , piece_id=LEKOWTLBNHVNH26VLGWUHPBQEOMRXVN3PPM4MPPKELYQLFCARF7Q, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.034894584 (536h20m53.905977629s) [noise=true , address=145.79.227.37:4567 , piece_id=BBCSD527H5AMFVJ3CIGEWETSAPHIUXATPXO7EUXE4TNMXFEAJLPQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035120204 (536h20m53.906203249s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035320546 (536h20m53.906403591s) [noise=true , address=145.79.227.48:4302 , piece_id=LXGVHZHN6N5G6HN57PDS5NCQYNVDQ66FOT2VBZUM5TWJMHY7AC6A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035736688 (536h20m53.906819733s) [noise=true , address=45.91.116.6:4975 , piece_id=PPCMP3EZKPSR3MKG5GLU6DBFNIIJOZ7QTNNKT6NFWITKTXJ7SQOQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.035750544 (536h20m53.906833579s) [noise=true , address=145.79.227.11:4730 , piece_id=ZKPTQGIZZAI3T42ZFZYKPXHGOFJEMA2LDUO5IA5ISYD4PT6ELTNQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.036308316 (536h20m53.907391361s) [noise=true , address=45.91.116.9:4264 , piece_id=YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.036941011 (536h20m53.908024066s) [noise=true , address=45.91.116.7:4009 , piece_id=3ZRK2J6F7NHLNF6FELNSF6RRXHECLH6GCWOBQVUX3O5JMDOX64KA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.037561823 (536h20m53.908644868s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.038148420 (536h20m53.909231465s) [noise=true , address=145.79.227.36:4518 , piece_id=BUE6DDX7IXJVZ3FWLEIMINXS5BOJSBQWQ64LLTVRJDHL6MIHOPUA, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:54.038187164 (536h20m53.909270199s) [noise=true , address=145.79.227.12:4767 , piece_id=D55M4EN5H5M3B2YHGDD6HKLVM3RDMTQUTCWWRVRL5ZGAWNR2B6JQ, seg_pos={0 0}, upload=470384]: Uploading piece... Dec 27 23:49:55.748361325 (536h20m55.61944436s) [upload=470384]: Next returned nil segment Dec 27 23:49:55.748375892 (536h20m55.619458937s) [upload=470384]: Waiting for error group managing segments... Dec 27 23:49:56.028135006 (536h20m55.899218051s) [noise=true , address=145.79.227.36:4518 , piece_id=BUE6DDX7IXJVZ3FWLEIMINXS5BOJSBQWQ64LLTVRJDHL6MIHOPUA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.039866044 (536h20m55.910949089s) [noise=true , address=145.79.227.11:4730 , piece_id=ZKPTQGIZZAI3T42ZFZYKPXHGOFJEMA2LDUO5IA5ISYD4PT6ELTNQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.046130612 (536h20m55.917213647s) [noise=true , address=145.79.227.37:4567 , piece_id=BBCSD527H5AMFVJ3CIGEWETSAPHIUXATPXO7EUXE4TNMXFEAJLPQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.047418615 (536h20m55.91850166s) [noise=true , address=145.79.227.5:4221 , piece_id=V6BJWTM5DDNW6GQGAH2A5LSP66V3KU4PBDQRWJKZ4Q7IHTSPRLTA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.053789636 (536h20m55.924872671s) [noise=true , address=145.79.227.25:4618 , piece_id=LEKOWTLBNHVNH26VLGWUHPBQEOMRXVN3PPM4MPPKELYQLFCARF7Q, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.059014726 (536h20m55.930097761s) [noise=true , address=145.79.227.48:4302 , piece_id=LXGVHZHN6N5G6HN57PDS5NCQYNVDQ66FOT2VBZUM5TWJMHY7AC6A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.065164977 (536h20m55.936248012s) [noise=true , address=145.79.227.12:4767 , piece_id=D55M4EN5H5M3B2YHGDD6HKLVM3RDMTQUTCWWRVRL5ZGAWNR2B6JQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.147749461 (536h20m56.018832496s) [noise=true , address=145.79.227.49:4337 , piece_id=BTOEY4JTYXYSHHPWNE4F3EFC222TP7CV5DETYXESMN3IHFWACA7A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.261321269 (536h20m56.132404314s) [noise=true , address=45.91.116.13:4155 , piece_id=E2PH5R64L7SJKFO6MIEOIQBY6RUQRZ7JWN2ZDMSJ4OPFWXVYW3HA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.320094221 (536h20m56.191177256s) [noise=true , address=45.91.116.9:4264 , piece_id=YOO5HGJY5A7M67BEGRRELNT3GKPA7TXKXVGFIFEX63XJGU22CR3A, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.320473874 (536h20m56.191556909s) [noise=true , address=45.91.116.6:4975 , piece_id=PPCMP3EZKPSR3MKG5GLU6DBFNIIJOZ7QTNNKT6NFWITKTXJ7SQOQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.343378848 (536h20m56.214461883s) [noise=true , address=45.91.116.7:4009 , piece_id=3ZRK2J6F7NHLNF6FELNSF6RRXHECLH6GCWOBQVUX3O5JMDOX64KA, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.972167347 (536h20m56.843250382s) [noise=true , address=45.91.116.8:4047 , piece_id=YLDG54NZBCROX7XKM2JN7NRZZWQ75GFQVDNAFKCMTXTHRY7CDXGQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: <nil> Dec 27 23:49:56.972188226 (536h20m56.843271261s) [seg_pos={0 0}, upload=470384]: Segment reached optimal threshold of 13 pieces. Dec 27 23:49:56.972491374 (536h20m56.843574409s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 23:49:56.972500511 (536h20m56.843583556s) [noise=true , address=77.90.31.21:4877 , piece_id=TE6LVWSKO6HF3PJU2J5TSJ7H4ZVQYZJ6HWI6MX4BTRTVS6GBGDGQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972537631 (536h20m56.843620676s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45248->145.79.227.51:4410: use of closed network connection Dec 27 23:49:56.972554634 (536h20m56.843637679s) [noise=true , address=145.79.227.51:4410 , piece_id=XXY62WLBXWJMFKERX4XO6SHSXHP6NELHHMJ64YGECAA7LZS2F6QQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972826402 (536h20m56.843909437s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50908->77.90.31.13:4627: use of closed network connection Dec 27 23:49:56.972835820 (536h20m56.843918875s) [noise=true , address=77.90.31.13:4627 , piece_id=FNAMLA2LJE7A3JYXKJYNPKUDH5RORBF6XYCWVLJVCQOU5SUUZKEQ, seg_pos={0 0}, upload=470384]: Long tail cancellation due to optimal threshold Dec 27 23:49:56.972856730 (536h20m56.843939775s) [seg_pos={0 0}, upload=470384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:49:56.972952071 (536h20m56.844035106s) [upload=470384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:49:56.972959264 (536h20m56.844042299s) [upload=470384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:49:57.260037595 (536h20m57.13112065s) [upload=470384]: Done uploading segments... Dec 27 23:49:57.260078192 (536h20m57.131161237s) [upload=470384]: Upload finished. err: <nil> Duration: 30.00298641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.557505416s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:50:17 UTC) orbiter warning: context deadline exceeded Duration: 3.55752848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.583505713s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:49:16 UTC) orbiter warning: context deadline exceeded Duration: 3.58353518s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)