Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 12.960571881s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:35:07 UTC) orbiter warning: context deadline exceeded Duration: 12.960601898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..X.XX.] duration: 6.112385579s trace: https://jaeger.dev.storj.io/trace/4f19139237f31d3e FAIL (2025-12-30 12:35:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ) context deadline exceeded ======LOGS====== 27.933554343s) [upload=523389]: Uploading segments... Dec 30 12:34:28.062530269 (597h5m27.933613304s) [upload=523389]: Got next segment. Inline: false Dec 30 12:34:28.062561899 (597h5m27.933644944s) [upload=523389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:34:28.062566488 (597h5m27.933649523s) [upload=523389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:34:28.473074067 (597h5m28.344157102s) [seg_pos={0 0}, upload=523389]: Begin upload segment... Dec 30 12:34:28.473117199 (597h5m28.344200244s) [seg_pos={0 0}, upload=523389]: Done begin upload segment. Dec 30 12:34:28.473147386 (597h5m28.344230431s) [noise=true , address=45.91.116.14:4192 , piece_id=WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.473179908 (597h5m28.344262953s) [noise=true , address=145.79.227.52:4446 , piece_id=5535UKKNNPJQN3726KYNLFJD6MAEOF3MJMCBP447A5F2NU5BHJ5A, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.473972898 (597h5m28.345055934s) [noise=true , address=145.79.227.2:4114 , piece_id=DV6GTPZXGNOKVSDOWWLYTGXD6MUGTCTNPOQ4J7ZCGD4UZKAIQLNA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474250327 (597h5m28.345333362s) [noise=true , address=145.79.227.37:4565 , piece_id=O7VT2EFMNTQPKWFEKNH45XSRGIA3XRBHPFFCE4XMGIWOFMICZFUA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474533125 (597h5m28.34561617s) [noise=true , address=77.90.31.10:4519 , piece_id=ND7QOTOMYVFZFEUL2QRFYMWWRNU4SD7OIR6C35ZKELN7EC4IHSGQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474840690 (597h5m28.345923735s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475243327 (597h5m28.346326362s) [noise=true , address=145.79.227.19:4078 , piece_id=C2FEBM4AUJ4CGPZHFHD4W52LK3QKOY623BHCXKATOREAOU4ZUVQA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475610336 (597h5m28.346693381s) [noise=true , address=145.79.227.14:4840 , piece_id=FTLJQJWJZFLVIDO37KUNQ4KCYPNSBKMMBCXH4WUK5N64EBMDQXAA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475962998 (597h5m28.347046043s) [noise=true , address=145.79.227.51:4409 , piece_id=3TTUNIKPXTEAWQP5XTPRF5OZW5XBRXI54DHYJD574O323MDVE2OQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.476350606 (597h5m28.347433651s) [noise=true , address=145.79.227.47:4267 , piece_id=ZWCZ3D4WVBRVCSU2PDEYPYDEISXUY6SNUAQTACVTZFPTV3ZD3JDQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478039471 (597h5m28.349122506s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478363007 (597h5m28.349446042s) [noise=true , address=145.79.227.16:4909 , piece_id=XTJZ2V45BNFKW23SXQKVHMEUQDKZ2KBY3Z2CBUGN2AOCIZA6F4RA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478669150 (597h5m28.349752185s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.480238398 (597h5m28.351321433s) [noise=true , address=145.79.227.41:4085 , piece_id=WSVOWMDYP6PJOOPXQFMVF5INVGURTDQU7XMWUIOJXO3CJBDBVNYQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.480942038 (597h5m28.352025083s) [noise=true , address=145.79.227.21:4473 , piece_id=TIEXWS4764HSI2G7ECZCRFHNX7LCIOYWNIY4JTRJDGUS4IS4C5ZA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.481400371 (597h5m28.352483416s) [noise=true , address=45.91.116.9:4264 , piece_id=RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:30.177689396 (597h5m30.048772441s) [upload=523389]: Next returned nil segment Dec 30 12:34:30.177725534 (597h5m30.048808589s) [upload=523389]: Waiting for error group managing segments... Dec 30 12:34:30.456354254 (597h5m30.32743729s) [noise=true , address=145.79.227.2:4114 , piece_id=DV6GTPZXGNOKVSDOWWLYTGXD6MUGTCTNPOQ4J7ZCGD4UZKAIQLNA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.463018855 (597h5m30.3341019s) [noise=true , address=145.79.227.14:4840 , piece_id=FTLJQJWJZFLVIDO37KUNQ4KCYPNSBKMMBCXH4WUK5N64EBMDQXAA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.464325704 (597h5m30.335408739s) [noise=true , address=145.79.227.41:4085 , piece_id=WSVOWMDYP6PJOOPXQFMVF5INVGURTDQU7XMWUIOJXO3CJBDBVNYQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.471056961 (597h5m30.342139996s) [noise=true , address=145.79.227.21:4473 , piece_id=TIEXWS4764HSI2G7ECZCRFHNX7LCIOYWNIY4JTRJDGUS4IS4C5ZA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.476503261 (597h5m30.347586306s) [noise=true , address=145.79.227.47:4267 , piece_id=ZWCZ3D4WVBRVCSU2PDEYPYDEISXUY6SNUAQTACVTZFPTV3ZD3JDQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.478188490 (597h5m30.349271535s) [noise=true , address=145.79.227.52:4446 , piece_id=5535UKKNNPJQN3726KYNLFJD6MAEOF3MJMCBP447A5F2NU5BHJ5A, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.479685932 (597h5m30.350768967s) [noise=true , address=145.79.227.51:4409 , piece_id=3TTUNIKPXTEAWQP5XTPRF5OZW5XBRXI54DHYJD574O323MDVE2OQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.481233178 (597h5m30.352316223s) [noise=true , address=145.79.227.37:4565 , piece_id=O7VT2EFMNTQPKWFEKNH45XSRGIA3XRBHPFFCE4XMGIWOFMICZFUA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.481491650 (597h5m30.352574695s) [noise=true , address=145.79.227.19:4078 , piece_id=C2FEBM4AUJ4CGPZHFHD4W52LK3QKOY623BHCXKATOREAOU4ZUVQA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.490243906 (597h5m30.361326951s) [noise=true , address=145.79.227.16:4909 , piece_id=XTJZ2V45BNFKW23SXQKVHMEUQDKZ2KBY3Z2CBUGN2AOCIZA6F4RA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.642508860 (597h5m30.513591905s) [noise=true , address=77.90.31.10:4519 , piece_id=ND7QOTOMYVFZFEUL2QRFYMWWRNU4SD7OIR6C35ZKELN7EC4IHSGQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.856480515 (597h5m30.72756357s) [noise=true , address=45.91.116.14:4192 , piece_id=WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:31.179232341 (597h5m31.050315376s) [noise=true , address=45.91.116.9:4264 , piece_id=RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:31.179259383 (597h5m31.050342428s) [seg_pos={0 0}, upload=523389]: Segment reached optimal threshold of 13 pieces. Dec 30 12:34:31.179546159 (597h5m31.050629194s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 12:34:31.179559124 (597h5m31.050642169s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180071329 (597h5m31.051154384s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection Dec 30 12:34:31.180082170 (597h5m31.051165215s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180229941 (597h5m31.051312986s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection Dec 30 12:34:31.180242074 (597h5m31.051325119s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180265309 (597h5m31.051348354s) [seg_pos={0 0}, upload=523389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:34:31.180353156 (597h5m31.051436201s) [upload=523389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:34:31.180361041 (597h5m31.051444076s) [upload=523389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:34:31.477826098 (597h5m31.348909133s) [upload=523389]: Done uploading segments... Dec 30 12:34:31.477844944 (597h5m31.348927989s) [upload=523389]: Upload finished. err: <nil> Duration: 6.112429993s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XXXXXXX] duration: 5.001192041s trace: https://jaeger.dev.storj.io/trace/207be3d12c99182a FAIL (2025-12-30 12:34:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001210146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..X.XX] duration: 30.004174739s trace: https://jaeger.dev.storj.io/trace/4f19139237f31d3e FAIL (2025-12-30 12:34:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ) context deadline exceeded ======LOGS====== 27.933554343s) [upload=523389]: Uploading segments... Dec 30 12:34:28.062530269 (597h5m27.933613304s) [upload=523389]: Got next segment. Inline: false Dec 30 12:34:28.062561899 (597h5m27.933644944s) [upload=523389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:34:28.062566488 (597h5m27.933649523s) [upload=523389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:34:28.473074067 (597h5m28.344157102s) [seg_pos={0 0}, upload=523389]: Begin upload segment... Dec 30 12:34:28.473117199 (597h5m28.344200244s) [seg_pos={0 0}, upload=523389]: Done begin upload segment. Dec 30 12:34:28.473147386 (597h5m28.344230431s) [noise=true , address=45.91.116.14:4192 , piece_id=WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.473179908 (597h5m28.344262953s) [noise=true , address=145.79.227.52:4446 , piece_id=5535UKKNNPJQN3726KYNLFJD6MAEOF3MJMCBP447A5F2NU5BHJ5A, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.473972898 (597h5m28.345055934s) [noise=true , address=145.79.227.2:4114 , piece_id=DV6GTPZXGNOKVSDOWWLYTGXD6MUGTCTNPOQ4J7ZCGD4UZKAIQLNA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474250327 (597h5m28.345333362s) [noise=true , address=145.79.227.37:4565 , piece_id=O7VT2EFMNTQPKWFEKNH45XSRGIA3XRBHPFFCE4XMGIWOFMICZFUA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474533125 (597h5m28.34561617s) [noise=true , address=77.90.31.10:4519 , piece_id=ND7QOTOMYVFZFEUL2QRFYMWWRNU4SD7OIR6C35ZKELN7EC4IHSGQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.474840690 (597h5m28.345923735s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475243327 (597h5m28.346326362s) [noise=true , address=145.79.227.19:4078 , piece_id=C2FEBM4AUJ4CGPZHFHD4W52LK3QKOY623BHCXKATOREAOU4ZUVQA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475610336 (597h5m28.346693381s) [noise=true , address=145.79.227.14:4840 , piece_id=FTLJQJWJZFLVIDO37KUNQ4KCYPNSBKMMBCXH4WUK5N64EBMDQXAA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.475962998 (597h5m28.347046043s) [noise=true , address=145.79.227.51:4409 , piece_id=3TTUNIKPXTEAWQP5XTPRF5OZW5XBRXI54DHYJD574O323MDVE2OQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.476350606 (597h5m28.347433651s) [noise=true , address=145.79.227.47:4267 , piece_id=ZWCZ3D4WVBRVCSU2PDEYPYDEISXUY6SNUAQTACVTZFPTV3ZD3JDQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478039471 (597h5m28.349122506s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478363007 (597h5m28.349446042s) [noise=true , address=145.79.227.16:4909 , piece_id=XTJZ2V45BNFKW23SXQKVHMEUQDKZ2KBY3Z2CBUGN2AOCIZA6F4RA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.478669150 (597h5m28.349752185s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.480238398 (597h5m28.351321433s) [noise=true , address=145.79.227.41:4085 , piece_id=WSVOWMDYP6PJOOPXQFMVF5INVGURTDQU7XMWUIOJXO3CJBDBVNYQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.480942038 (597h5m28.352025083s) [noise=true , address=145.79.227.21:4473 , piece_id=TIEXWS4764HSI2G7ECZCRFHNX7LCIOYWNIY4JTRJDGUS4IS4C5ZA, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:28.481400371 (597h5m28.352483416s) [noise=true , address=45.91.116.9:4264 , piece_id=RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ, seg_pos={0 0}, upload=523389]: Uploading piece... Dec 30 12:34:30.177689396 (597h5m30.048772441s) [upload=523389]: Next returned nil segment Dec 30 12:34:30.177725534 (597h5m30.048808589s) [upload=523389]: Waiting for error group managing segments... Dec 30 12:34:30.456354254 (597h5m30.32743729s) [noise=true , address=145.79.227.2:4114 , piece_id=DV6GTPZXGNOKVSDOWWLYTGXD6MUGTCTNPOQ4J7ZCGD4UZKAIQLNA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.463018855 (597h5m30.3341019s) [noise=true , address=145.79.227.14:4840 , piece_id=FTLJQJWJZFLVIDO37KUNQ4KCYPNSBKMMBCXH4WUK5N64EBMDQXAA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.464325704 (597h5m30.335408739s) [noise=true , address=145.79.227.41:4085 , piece_id=WSVOWMDYP6PJOOPXQFMVF5INVGURTDQU7XMWUIOJXO3CJBDBVNYQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.471056961 (597h5m30.342139996s) [noise=true , address=145.79.227.21:4473 , piece_id=TIEXWS4764HSI2G7ECZCRFHNX7LCIOYWNIY4JTRJDGUS4IS4C5ZA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.476503261 (597h5m30.347586306s) [noise=true , address=145.79.227.47:4267 , piece_id=ZWCZ3D4WVBRVCSU2PDEYPYDEISXUY6SNUAQTACVTZFPTV3ZD3JDQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.478188490 (597h5m30.349271535s) [noise=true , address=145.79.227.52:4446 , piece_id=5535UKKNNPJQN3726KYNLFJD6MAEOF3MJMCBP447A5F2NU5BHJ5A, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.479685932 (597h5m30.350768967s) [noise=true , address=145.79.227.51:4409 , piece_id=3TTUNIKPXTEAWQP5XTPRF5OZW5XBRXI54DHYJD574O323MDVE2OQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.481233178 (597h5m30.352316223s) [noise=true , address=145.79.227.37:4565 , piece_id=O7VT2EFMNTQPKWFEKNH45XSRGIA3XRBHPFFCE4XMGIWOFMICZFUA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.481491650 (597h5m30.352574695s) [noise=true , address=145.79.227.19:4078 , piece_id=C2FEBM4AUJ4CGPZHFHD4W52LK3QKOY623BHCXKATOREAOU4ZUVQA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.490243906 (597h5m30.361326951s) [noise=true , address=145.79.227.16:4909 , piece_id=XTJZ2V45BNFKW23SXQKVHMEUQDKZ2KBY3Z2CBUGN2AOCIZA6F4RA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.642508860 (597h5m30.513591905s) [noise=true , address=77.90.31.10:4519 , piece_id=ND7QOTOMYVFZFEUL2QRFYMWWRNU4SD7OIR6C35ZKELN7EC4IHSGQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:30.856480515 (597h5m30.72756357s) [noise=true , address=45.91.116.14:4192 , piece_id=WOTABYUDXRVNI6TZUAUMMBEHOMI2IA6MS6KLKM2MJFZAWSAAEAGA, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:31.179232341 (597h5m31.050315376s) [noise=true , address=45.91.116.9:4264 , piece_id=RDRJTWDTMEW763JVRBQOBJVBUJSQQDHMX5UYFZUVUMHTHGJI76WQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: <nil> Dec 30 12:34:31.179259383 (597h5m31.050342428s) [seg_pos={0 0}, upload=523389]: Segment reached optimal threshold of 13 pieces. Dec 30 12:34:31.179546159 (597h5m31.050629194s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 12:34:31.179559124 (597h5m31.050642169s) [noise=true , address=45.91.116.10:4081 , piece_id=TLWMOF6GW5ZZ56GPNYGPYQKLVUL2PFABT5PCI3AP76RLZTITAD5A, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180071329 (597h5m31.051154384s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44270->45.91.116.6:4976: use of closed network connection Dec 30 12:34:31.180082170 (597h5m31.051165215s) [noise=true , address=45.91.116.6:4976 , piece_id=DYJT2GIWJYNY6BJW6OPVTRWAWZEXR3QPGR7HRSMFFLUT6BBMYF4Q, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180229941 (597h5m31.051312986s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.13:4803: use of closed network connection Dec 30 12:34:31.180242074 (597h5m31.051325119s) [noise=true , address=145.79.227.13:4803 , piece_id=O65TWFJU4N3HMGNY4G57GKWMDBRSMXRQROX33XYSAK2LIBGQP3IQ, seg_pos={0 0}, upload=523389]: Long tail cancellation due to optimal threshold Dec 30 12:34:31.180265309 (597h5m31.051348354s) [seg_pos={0 0}, upload=523389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:34:31.180353156 (597h5m31.051436201s) [upload=523389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:34:31.180361041 (597h5m31.051444076s) [upload=523389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:34:31.477826098 (597h5m31.348909133s) [upload=523389]: Done uploading segments... Dec 30 12:34:31.477844944 (597h5m31.348927989s) [upload=523389]: Upload finished. err: <nil> Duration: 30.004210407s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 14.559167257s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:34:42 UTC) orbiter warning: context deadline exceeded Duration: 14.559206942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.................X..................X.....X.X...X....] duration: 45.329799288s trace: https://jaeger.dev.storj.io/trace/12ab50cae54c34b8 FAIL (2025-12-30 12:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3335]: Uploading piece... Dec 30 12:30:35.776704210 (597h1m35.647787255s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777174075 (597h1m35.64825712s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777679076 (597h1m35.648762121s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778158129 (597h1m35.649241164s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778618795 (597h1m35.649701841s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779090805 (597h1m35.65017385s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779577592 (597h1m35.650660637s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.780567807 (597h1m35.651650842s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.781941372 (597h1m35.653024417s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.783206401 (597h1m35.654289436s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.784920044 (597h1m35.656003089s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.786933338 (597h1m35.658016383s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.788728716 (597h1m35.659811751s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.790109045 (597h1m35.66119208s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.791393441 (597h1m35.662476486s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.792676213 (597h1m35.663759258s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:37.211774620 (597h1m37.082857655s) [upload=523335]: Next returned nil segment Dec 30 12:30:37.211791042 (597h1m37.082874077s) [upload=523335]: Waiting for error group managing segments... Dec 30 12:30:37.571833998 (597h1m37.442917033s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632153686 (597h1m37.503236721s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632641054 (597h1m37.503724089s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.645908035 (597h1m37.516991081s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.655521531 (597h1m37.526604576s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.719473996 (597h1m37.590557042s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.897438869 (597h1m37.768521904s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.945885809 (597h1m37.816968854s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.988490013 (597h1m37.859573048s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.499362532 (597h1m38.370445567s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.542379102 (597h1m38.413462138s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.568219716 (597h1m38.439302761s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.648331221 (597h1m38.519414256s) [noise=true , address=145.79.227.35:4512 , piece_id=6SWJYOSFS3QLAP3LFZV5G66EAINO5AXBWUZBMNZENWIHWVSCUCEA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659794188 (597h1m38.530877233s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659818734 (597h1m38.530901769s) [seg_pos={0 0}, upload=523335]: Segment reached optimal threshold of 14 pieces. Dec 30 12:30:38.660086344 (597h1m38.531169389s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660094008 (597h1m38.531177043s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660405782 (597h1m38.531488827s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection Dec 30 12:30:38.660413647 (597h1m38.531496702s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660430640 (597h1m38.531513675s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): internal: context canceled Dec 30 12:30:38.660470045 (597h1m38.53155308s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660561720 (597h1m38.531644765s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660569023 (597h1m38.531652068s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660606895 (597h1m38.53168994s) [seg_pos={0 0}, upload=523335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:30:38.660701275 (597h1m38.53178432s) [upload=523335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:38.660707267 (597h1m38.531790312s) [upload=523335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:38.952034268 (597h1m38.823117323s) [upload=523335]: Done uploading segments... Dec 30 12:30:38.952059425 (597h1m38.82314247s) [upload=523335]: Upload finished. err: <nil> Duration: 45.329828704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..X.X] duration: 30.001862918s trace: https://jaeger.dev.storj.io/trace/2600a499ffd7b6b2 FAIL (2025-12-30 12:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: KG6GWUVCBF7FLKNSPP6FQT73ZYWVG4SAHLOCOM6IISKYO5SRQ6RA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: BMJONH472BY2LDKREC2EOQG7YV63E2WEBUPQHCFIOWLYRULVOHZQ) context deadline exceeded ======LOGS====== Dec 30 12:33:46.899146049 (597h4m46.770229084s) [upload=523381]: Starting upload Dec 30 12:33:46.899200343 (597h4m46.770283378s) [upload=523381]: Done starting upload Dec 30 12:33:46.899325201 (597h4m46.770408236s) [upload=523381]: Uploading segments... Dec 30 12:33:46.899430732 (597h4m46.770513767s) [upload=523381]: Got next segment. Inline: false Dec 30 12:33:46.899470939 (597h4m46.770553974s) [upload=523381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:33:46.899475928 (597h4m46.770558963s) [upload=523381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:33:47.147625692 (597h4m47.018708738s) [seg_pos={0 0}, upload=523381]: Begin upload segment... Dec 30 12:33:47.147695125 (597h4m47.01877817s) [seg_pos={0 0}, upload=523381]: Done begin upload segment. Dec 30 12:33:47.147792550 (597h4m47.018875585s) [noise=true , address=145.79.227.11:4729 , piece_id=UFOQ5X73VPA6XYROL7TJML5HJYPSUNLZLJ2GJAFIC5YX4MFMXV2A, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.147807979 (597h4m47.018891014s) [noise=true , address=145.79.227.21:4475 , piece_id=AY5HCDWXE6DXHM4JOMKQ2KKT2L4SC7PR2GRIKIBSZMDYCTAHC7RA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.147871280 (597h4m47.018954315s) [noise=true , address=77.90.31.14:4662 , piece_id=BMJONH472BY2LDKREC2EOQG7YV63E2WEBUPQHCFIOWLYRULVOHZQ, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.148651085 (597h4m47.01973412s) [noise=true , address=145.79.227.26:4654 , piece_id=CDYN6VZR6V7XL5UAVJGIFCJEJYKRJE4A2CINZPANOBBXTCA72TJQ, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.148959572 (597h4m47.020042607s) [noise=true , address=45.91.116.11:4117 , piece_id=RP2LRISOYJYDV24XYKKI5ROBQQOBB4P2CZV4PGL7AICFXL5PPREQ, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.149222954 (597h4m47.020305989s) [noise=true , address=145.79.227.19:4077 , piece_id=XDCCLSCRDNOMCQUH3X54HKLRA5ASEELP5FJPYFD5ZNCA33GVDZDA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.149544236 (597h4m47.020627271s) [noise=true , address=145.79.227.46:4602 , piece_id=K6DBHZEOWGY3FBMQXLKTBJ25WXTAVFI6HWGHSKJS4G5R5MBOQ4MA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.150656333 (597h4m47.021739368s) [noise=true , address=45.91.116.7:4012 , piece_id=KXO76KTQ73SDMRBHALBUU5FZYHLS4LST63Y6KDPXSZE5Z266T34Q, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.152075405 (597h4m47.02315845s) [noise=true , address=77.90.31.11:4553 , piece_id=KG6GWUVCBF7FLKNSPP6FQT73ZYWVG4SAHLOCOM6IISKYO5SRQ6RA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.153288535 (597h4m47.02437157s) [noise=true , address=145.79.227.50:4373 , piece_id=KEVTUHXKTIJ4LCUCDTP5P35CTG55DP4NYSTK2KF5KOB5EMIHQEEA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.154500683 (597h4m47.025583718s) [noise=true , address=145.79.227.4:4187 , piece_id=SHTQJRS6LIF5VHTVNNEDGPJFWFDISCME4HIUII4KRB6ZNNWFEBVA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.155700779 (597h4m47.026783814s) [noise=true , address=145.79.227.3:4150 , piece_id=3PBKBULQZQFOWYCPPKZ2QXLK3S53GJWBW2DZPUCJI7SPNTZZMQZA, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.156756218 (597h4m47.027839253s) [noise=true , address=145.79.227.33:4869 , piece_id=BOUW4ZBZT3OZFZXQ4JZMFBKQA2UYEUZTNK4VJOMIBG36ZLINXGAQ, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.157032484 (597h4m47.028115519s) [noise=true , address=145.79.227.23:4547 , piece_id=W2TMTCGXA3SXYGAWB4HFS35L7GY3DKNJQMXTDB3IQCJMYEYAIR6Q, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.158157316 (597h4m47.029240351s) [noise=true , address=45.91.116.10:4084 , piece_id=F3IC6KBAJ3MGI3OKCPQ5DFDKEE5DUTI6H6S7ELJXHYYJTNHLQONQ, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:47.158576604 (597h4m47.029659639s) [noise=true , address=145.79.227.9:4366 , piece_id=S63RH6ZY6SYOXR7ZTARTNQULSDRKYD6Z5QIGXIUTM4O5KM646W6A, seg_pos={0 0}, upload=523381]: Uploading piece... Dec 30 12:33:48.839374112 (597h4m48.710457157s) [upload=523381]: Next returned nil segment Dec 30 12:33:48.839388620 (597h4m48.710471665s) [upload=523381]: Waiting for error group managing segments... Dec 30 12:33:49.121426913 (597h4m48.992509958s) [noise=true , address=145.79.227.19:4077 , piece_id=XDCCLSCRDNOMCQUH3X54HKLRA5ASEELP5FJPYFD5ZNCA33GVDZDA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.131248194 (597h4m49.002331239s) [noise=true , address=145.79.227.4:4187 , piece_id=SHTQJRS6LIF5VHTVNNEDGPJFWFDISCME4HIUII4KRB6ZNNWFEBVA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.141969218 (597h4m49.013052253s) [noise=true , address=145.79.227.21:4475 , piece_id=AY5HCDWXE6DXHM4JOMKQ2KKT2L4SC7PR2GRIKIBSZMDYCTAHC7RA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.150382919 (597h4m49.021465954s) [noise=true , address=145.79.227.50:4373 , piece_id=KEVTUHXKTIJ4LCUCDTP5P35CTG55DP4NYSTK2KF5KOB5EMIHQEEA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.150632504 (597h4m49.021715539s) [noise=true , address=145.79.227.23:4547 , piece_id=W2TMTCGXA3SXYGAWB4HFS35L7GY3DKNJQMXTDB3IQCJMYEYAIR6Q, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.161762186 (597h4m49.032845232s) [noise=true , address=145.79.227.46:4602 , piece_id=K6DBHZEOWGY3FBMQXLKTBJ25WXTAVFI6HWGHSKJS4G5R5MBOQ4MA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.162500323 (597h4m49.033583368s) [noise=true , address=145.79.227.26:4654 , piece_id=CDYN6VZR6V7XL5UAVJGIFCJEJYKRJE4A2CINZPANOBBXTCA72TJQ, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.163405146 (597h4m49.034488181s) [noise=true , address=145.79.227.11:4729 , piece_id=UFOQ5X73VPA6XYROL7TJML5HJYPSUNLZLJ2GJAFIC5YX4MFMXV2A, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.170229470 (597h4m49.041312505s) [noise=true , address=145.79.227.9:4366 , piece_id=S63RH6ZY6SYOXR7ZTARTNQULSDRKYD6Z5QIGXIUTM4O5KM646W6A, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.232166317 (597h4m49.103249352s) [noise=true , address=145.79.227.3:4150 , piece_id=3PBKBULQZQFOWYCPPKZ2QXLK3S53GJWBW2DZPUCJI7SPNTZZMQZA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.322444800 (597h4m49.193527835s) [noise=true , address=77.90.31.11:4553 , piece_id=KG6GWUVCBF7FLKNSPP6FQT73ZYWVG4SAHLOCOM6IISKYO5SRQ6RA, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.600334845 (597h4m49.47141788s) [noise=true , address=45.91.116.10:4084 , piece_id=F3IC6KBAJ3MGI3OKCPQ5DFDKEE5DUTI6H6S7ELJXHYYJTNHLQONQ, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.631313513 (597h4m49.502396558s) [noise=true , address=77.90.31.14:4662 , piece_id=BMJONH472BY2LDKREC2EOQG7YV63E2WEBUPQHCFIOWLYRULVOHZQ, seg_pos={0 0}, upload=523381]: Done uploading piece. err: <nil> Dec 30 12:33:49.631338911 (597h4m49.502421956s) [seg_pos={0 0}, upload=523381]: Segment reached optimal threshold of 13 pieces. Dec 30 12:33:49.631646246 (597h4m49.502729301s) [noise=true , address=45.91.116.11:4117 , piece_id=RP2LRISOYJYDV24XYKKI5ROBQQOBB4P2CZV4PGL7AICFXL5PPREQ, seg_pos={0 0}, upload=523381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:51768->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:33:49.631656406 (597h4m49.502739441s) [noise=true , address=45.91.116.11:4117 , piece_id=RP2LRISOYJYDV24XYKKI5ROBQQOBB4P2CZV4PGL7AICFXL5PPREQ, seg_pos={0 0}, upload=523381]: Long tail cancellation due to optimal threshold Dec 30 12:33:49.631667527 (597h4m49.502750562s) [noise=true , address=45.91.116.7:4012 , piece_id=KXO76KTQ73SDMRBHALBUU5FZYHLS4LST63Y6KDPXSZE5Z266T34Q, seg_pos={0 0}, upload=523381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 12:33:49.631673768 (597h4m49.502756823s) [noise=true , address=45.91.116.7:4012 , piece_id=KXO76KTQ73SDMRBHALBUU5FZYHLS4LST63Y6KDPXSZE5Z266T34Q, seg_pos={0 0}, upload=523381]: Long tail cancellation due to optimal threshold Dec 30 12:33:49.631822491 (597h4m49.502905546s) [noise=true , address=145.79.227.33:4869 , piece_id=BOUW4ZBZT3OZFZXQ4JZMFBKQA2UYEUZTNK4VJOMIBG36ZLINXGAQ, seg_pos={0 0}, upload=523381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 12:33:49.631830537 (597h4m49.502913572s) [noise=true , address=145.79.227.33:4869 , piece_id=BOUW4ZBZT3OZFZXQ4JZMFBKQA2UYEUZTNK4VJOMIBG36ZLINXGAQ, seg_pos={0 0}, upload=523381]: Long tail cancellation due to optimal threshold Dec 30 12:33:49.631955284 (597h4m49.503038329s) [seg_pos={0 0}, upload=523381]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:33:49.632054113 (597h4m49.503137148s) [upload=523381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:33:49.632063370 (597h4m49.503146415s) [upload=523381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:33:49.924778674 (597h4m49.795861719s) [upload=523381]: Done uploading segments... Dec 30 12:33:49.924800055 (597h4m49.7958831s) [upload=523381]: Upload finished. err: <nil> Duration: 30.001878667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XXXXXX] duration: 5.001761794s trace: https://jaeger.dev.storj.io/trace/aebb58c908568ae FAIL (2025-12-30 12:34:11 UTC) orbiter warning: context deadline exceeded Duration: 5.00178062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.471569003s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:33:47 UTC) orbiter warning: context deadline exceeded Duration: 6.471581829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 24.800427549s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:33:40 UTC) orbiter warning: context deadline exceeded Duration: 24.800452976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.................X..................X.....X.X...X...] duration: 40.246113631s trace: https://jaeger.dev.storj.io/trace/12ab50cae54c34b8 FAIL (2025-12-30 12:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3335]: Uploading piece... Dec 30 12:30:35.776704210 (597h1m35.647787255s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777174075 (597h1m35.64825712s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777679076 (597h1m35.648762121s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778158129 (597h1m35.649241164s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778618795 (597h1m35.649701841s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779090805 (597h1m35.65017385s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779577592 (597h1m35.650660637s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.780567807 (597h1m35.651650842s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.781941372 (597h1m35.653024417s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.783206401 (597h1m35.654289436s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.784920044 (597h1m35.656003089s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.786933338 (597h1m35.658016383s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.788728716 (597h1m35.659811751s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.790109045 (597h1m35.66119208s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.791393441 (597h1m35.662476486s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.792676213 (597h1m35.663759258s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:37.211774620 (597h1m37.082857655s) [upload=523335]: Next returned nil segment Dec 30 12:30:37.211791042 (597h1m37.082874077s) [upload=523335]: Waiting for error group managing segments... Dec 30 12:30:37.571833998 (597h1m37.442917033s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632153686 (597h1m37.503236721s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632641054 (597h1m37.503724089s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.645908035 (597h1m37.516991081s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.655521531 (597h1m37.526604576s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.719473996 (597h1m37.590557042s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.897438869 (597h1m37.768521904s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.945885809 (597h1m37.816968854s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.988490013 (597h1m37.859573048s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.499362532 (597h1m38.370445567s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.542379102 (597h1m38.413462138s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.568219716 (597h1m38.439302761s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.648331221 (597h1m38.519414256s) [noise=true , address=145.79.227.35:4512 , piece_id=6SWJYOSFS3QLAP3LFZV5G66EAINO5AXBWUZBMNZENWIHWVSCUCEA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659794188 (597h1m38.530877233s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659818734 (597h1m38.530901769s) [seg_pos={0 0}, upload=523335]: Segment reached optimal threshold of 14 pieces. Dec 30 12:30:38.660086344 (597h1m38.531169389s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660094008 (597h1m38.531177043s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660405782 (597h1m38.531488827s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection Dec 30 12:30:38.660413647 (597h1m38.531496702s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660430640 (597h1m38.531513675s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): internal: context canceled Dec 30 12:30:38.660470045 (597h1m38.53155308s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660561720 (597h1m38.531644765s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660569023 (597h1m38.531652068s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660606895 (597h1m38.53168994s) [seg_pos={0 0}, upload=523335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:30:38.660701275 (597h1m38.53178432s) [upload=523335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:38.660707267 (597h1m38.531790312s) [upload=523335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:38.952034268 (597h1m38.823117323s) [upload=523335]: Done uploading segments... Dec 30 12:30:38.952059425 (597h1m38.82314247s) [upload=523335]: Upload finished. err: <nil> Duration: 40.246141364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..X.] duration: 6.202229252s trace: https://jaeger.dev.storj.io/trace/4bdcd05ae3182b5e FAIL (2025-12-30 12:33:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA) context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Dec 30 12:32:12.883854427 (597h3m12.754937482s) [upload=523359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:32:13.131825261 (597h3m13.002908296s) [seg_pos={0 0}, upload=523359]: Begin upload segment... Dec 30 12:32:13.131877831 (597h3m13.002960866s) [seg_pos={0 0}, upload=523359]: Done begin upload segment. Dec 30 12:32:13.131923719 (597h3m13.003006764s) [noise=true , address=45.91.116.8:4045 , piece_id=BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.131954999 (597h3m13.003038034s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132391690 (597h3m13.003474725s) [noise=true , address=77.90.31.17:4773 , piece_id=XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132670470 (597h3m13.003753505s) [noise=true , address=145.79.227.14:4837 , piece_id=356H4GRGOSGRZQ5YATOJHA77CHNDRBQ5LH2HEZNFLPSQTMXHH2BA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132955142 (597h3m13.004038177s) [noise=true , address=77.90.31.11:4554 , piece_id=3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.133566527 (597h3m13.004649572s) [noise=true , address=145.79.227.9:4365 , piece_id=CYX66KBGPQ6INE6Q4C5Y66IWJB6NRYNXAAKVSOOMSNT35AUDLIPA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134010783 (597h3m13.005093828s) [noise=true , address=77.90.31.12:4590 , piece_id=WFHDEJEDWKCPXF64FDIWAZ35QHZAKYYX2JLIACE2NS3PQHOHLJZA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134341122 (597h3m13.005424157s) [noise=true , address=145.79.227.13:4804 , piece_id=RIR4H2QD43SFG7XM566BJQ3FDNPEQSDUK372P4DYVWYRG26NIP4A, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134678695 (597h3m13.00576173s) [noise=true , address=145.79.227.47:4267 , piece_id=R4ZTPFWP3XIIKOGJ7XVN6WZH2RRTZCA2WASDV33K36ZPOOBAZ7AQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.135098845 (597h3m13.00618189s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.136280635 (597h3m13.00736367s) [noise=true , address=145.79.227.41:4087 , piece_id=44F3EC7ZJMSU67MYKRYCY5UQO6HY6F3F2SV5VYHVZGZDZVVMQSJQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.137735636 (597h3m13.008818671s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.138956881 (597h3m13.010039926s) [noise=true , address=145.79.227.46:4602 , piece_id=GDVL75E7RKKVTFLBXHG62ATXTTM47Q3FMY55T27QWJC6D45Y5BPA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.140057837 (597h3m13.011140882s) [noise=true , address=45.91.116.12:4226 , piece_id=4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.141256028 (597h3m13.012339073s) [noise=true , address=145.79.227.20:4438 , piece_id=PXTOH7Y7ABJX2ZBM5YHTFNLMFVOJCCDKUVZQOZMMZJDM4YNFSNQQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.142160110 (597h3m13.013243145s) [noise=true , address=145.79.227.3:4151 , piece_id=OR3HCZMRMCCPRXF7BYSOBJGYGUJSVDD5XXXSFDRP6ZI46N6GQBGA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:14.856224368 (597h3m14.727307404s) [upload=523359]: Next returned nil segment Dec 30 12:32:14.856236812 (597h3m14.727319857s) [upload=523359]: Waiting for error group managing segments... Dec 30 12:32:15.130990464 (597h3m15.002073499s) [noise=true , address=145.79.227.46:4602 , piece_id=GDVL75E7RKKVTFLBXHG62ATXTTM47Q3FMY55T27QWJC6D45Y5BPA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.131687191 (597h3m15.002770236s) [noise=true , address=145.79.227.41:4087 , piece_id=44F3EC7ZJMSU67MYKRYCY5UQO6HY6F3F2SV5VYHVZGZDZVVMQSJQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.141779318 (597h3m15.012862353s) [noise=true , address=145.79.227.13:4804 , piece_id=RIR4H2QD43SFG7XM566BJQ3FDNPEQSDUK372P4DYVWYRG26NIP4A, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.151289698 (597h3m15.022372743s) [noise=true , address=145.79.227.14:4837 , piece_id=356H4GRGOSGRZQ5YATOJHA77CHNDRBQ5LH2HEZNFLPSQTMXHH2BA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.151843141 (597h3m15.022926177s) [noise=true , address=145.79.227.20:4438 , piece_id=PXTOH7Y7ABJX2ZBM5YHTFNLMFVOJCCDKUVZQOZMMZJDM4YNFSNQQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.155563436 (597h3m15.026646481s) [noise=true , address=145.79.227.3:4151 , piece_id=OR3HCZMRMCCPRXF7BYSOBJGYGUJSVDD5XXXSFDRP6ZI46N6GQBGA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.160190236 (597h3m15.031273281s) [noise=true , address=145.79.227.9:4365 , piece_id=CYX66KBGPQ6INE6Q4C5Y66IWJB6NRYNXAAKVSOOMSNT35AUDLIPA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.164178039 (597h3m15.035261084s) [noise=true , address=145.79.227.47:4267 , piece_id=R4ZTPFWP3XIIKOGJ7XVN6WZH2RRTZCA2WASDV33K36ZPOOBAZ7AQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.378427733 (597h3m15.249510768s) [noise=true , address=77.90.31.11:4554 , piece_id=3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.421243280 (597h3m15.292326315s) [noise=true , address=77.90.31.17:4773 , piece_id=XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.434260938 (597h3m15.305343973s) [noise=true , address=77.90.31.12:4590 , piece_id=WFHDEJEDWKCPXF64FDIWAZ35QHZAKYYX2JLIACE2NS3PQHOHLJZA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.446578473 (597h3m15.317661508s) [noise=true , address=45.91.116.8:4045 , piece_id=BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.549935580 (597h3m15.421018615s) [noise=true , address=45.91.116.12:4226 , piece_id=4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.549960006 (597h3m15.421043051s) [seg_pos={0 0}, upload=523359]: Segment reached optimal threshold of 13 pieces. Dec 30 12:32:15.550163925 (597h3m15.42124697s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection Dec 30 12:32:15.550174265 (597h3m15.4212573s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550184334 (597h3m15.421267379s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection Dec 30 12:32:15.550190425 (597h3m15.42127347s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550484224 (597h3m15.421567259s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection Dec 30 12:32:15.550493433 (597h3m15.421576478s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550516155 (597h3m15.42159919s) [seg_pos={0 0}, upload=523359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:32:15.550669267 (597h3m15.421752312s) [upload=523359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:32:15.550677433 (597h3m15.421760468s) [upload=523359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:32:15.845600259 (597h3m15.716683304s) [upload=523359]: Done uploading segments... Dec 30 12:32:15.845624025 (597h3m15.71670706s) [upload=523359]: Upload finished. err: <nil> Duration: 6.202256525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XXXXX] duration: 5.000924587s trace: https://jaeger.dev.storj.io/trace/32d2ca6e05fadd04 FAIL (2025-12-30 12:32:57 UTC) orbiter warning: context deadline exceeded Duration: 5.000948943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 13.03953731s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:32:56 UTC) orbiter warning: context deadline exceeded Duration: 13.039551987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.................X..................X.....X.X...X..] duration: 34.631045377s trace: https://jaeger.dev.storj.io/trace/12ab50cae54c34b8 FAIL (2025-12-30 12:32:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3335]: Uploading piece... Dec 30 12:30:35.776704210 (597h1m35.647787255s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777174075 (597h1m35.64825712s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777679076 (597h1m35.648762121s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778158129 (597h1m35.649241164s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778618795 (597h1m35.649701841s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779090805 (597h1m35.65017385s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779577592 (597h1m35.650660637s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.780567807 (597h1m35.651650842s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.781941372 (597h1m35.653024417s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.783206401 (597h1m35.654289436s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.784920044 (597h1m35.656003089s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.786933338 (597h1m35.658016383s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.788728716 (597h1m35.659811751s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.790109045 (597h1m35.66119208s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.791393441 (597h1m35.662476486s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.792676213 (597h1m35.663759258s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:37.211774620 (597h1m37.082857655s) [upload=523335]: Next returned nil segment Dec 30 12:30:37.211791042 (597h1m37.082874077s) [upload=523335]: Waiting for error group managing segments... Dec 30 12:30:37.571833998 (597h1m37.442917033s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632153686 (597h1m37.503236721s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632641054 (597h1m37.503724089s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.645908035 (597h1m37.516991081s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.655521531 (597h1m37.526604576s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.719473996 (597h1m37.590557042s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.897438869 (597h1m37.768521904s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.945885809 (597h1m37.816968854s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.988490013 (597h1m37.859573048s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.499362532 (597h1m38.370445567s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.542379102 (597h1m38.413462138s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.568219716 (597h1m38.439302761s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.648331221 (597h1m38.519414256s) [noise=true , address=145.79.227.35:4512 , piece_id=6SWJYOSFS3QLAP3LFZV5G66EAINO5AXBWUZBMNZENWIHWVSCUCEA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659794188 (597h1m38.530877233s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659818734 (597h1m38.530901769s) [seg_pos={0 0}, upload=523335]: Segment reached optimal threshold of 14 pieces. Dec 30 12:30:38.660086344 (597h1m38.531169389s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660094008 (597h1m38.531177043s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660405782 (597h1m38.531488827s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection Dec 30 12:30:38.660413647 (597h1m38.531496702s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660430640 (597h1m38.531513675s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): internal: context canceled Dec 30 12:30:38.660470045 (597h1m38.53155308s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660561720 (597h1m38.531644765s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660569023 (597h1m38.531652068s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660606895 (597h1m38.53168994s) [seg_pos={0 0}, upload=523335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:30:38.660701275 (597h1m38.53178432s) [upload=523335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:38.660707267 (597h1m38.531790312s) [upload=523335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:38.952034268 (597h1m38.823117323s) [upload=523335]: Done uploading segments... Dec 30 12:30:38.952059425 (597h1m38.82314247s) [upload=523335]: Upload finished. err: <nil> Duration: 34.631065347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000890004s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:32:43 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000916064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..X] duration: 30.002317931s trace: https://jaeger.dev.storj.io/trace/4bdcd05ae3182b5e FAIL (2025-12-30 12:32:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA) context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Dec 30 12:32:12.883854427 (597h3m12.754937482s) [upload=523359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:32:13.131825261 (597h3m13.002908296s) [seg_pos={0 0}, upload=523359]: Begin upload segment... Dec 30 12:32:13.131877831 (597h3m13.002960866s) [seg_pos={0 0}, upload=523359]: Done begin upload segment. Dec 30 12:32:13.131923719 (597h3m13.003006764s) [noise=true , address=45.91.116.8:4045 , piece_id=BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.131954999 (597h3m13.003038034s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132391690 (597h3m13.003474725s) [noise=true , address=77.90.31.17:4773 , piece_id=XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132670470 (597h3m13.003753505s) [noise=true , address=145.79.227.14:4837 , piece_id=356H4GRGOSGRZQ5YATOJHA77CHNDRBQ5LH2HEZNFLPSQTMXHH2BA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.132955142 (597h3m13.004038177s) [noise=true , address=77.90.31.11:4554 , piece_id=3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.133566527 (597h3m13.004649572s) [noise=true , address=145.79.227.9:4365 , piece_id=CYX66KBGPQ6INE6Q4C5Y66IWJB6NRYNXAAKVSOOMSNT35AUDLIPA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134010783 (597h3m13.005093828s) [noise=true , address=77.90.31.12:4590 , piece_id=WFHDEJEDWKCPXF64FDIWAZ35QHZAKYYX2JLIACE2NS3PQHOHLJZA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134341122 (597h3m13.005424157s) [noise=true , address=145.79.227.13:4804 , piece_id=RIR4H2QD43SFG7XM566BJQ3FDNPEQSDUK372P4DYVWYRG26NIP4A, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.134678695 (597h3m13.00576173s) [noise=true , address=145.79.227.47:4267 , piece_id=R4ZTPFWP3XIIKOGJ7XVN6WZH2RRTZCA2WASDV33K36ZPOOBAZ7AQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.135098845 (597h3m13.00618189s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.136280635 (597h3m13.00736367s) [noise=true , address=145.79.227.41:4087 , piece_id=44F3EC7ZJMSU67MYKRYCY5UQO6HY6F3F2SV5VYHVZGZDZVVMQSJQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.137735636 (597h3m13.008818671s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.138956881 (597h3m13.010039926s) [noise=true , address=145.79.227.46:4602 , piece_id=GDVL75E7RKKVTFLBXHG62ATXTTM47Q3FMY55T27QWJC6D45Y5BPA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.140057837 (597h3m13.011140882s) [noise=true , address=45.91.116.12:4226 , piece_id=4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.141256028 (597h3m13.012339073s) [noise=true , address=145.79.227.20:4438 , piece_id=PXTOH7Y7ABJX2ZBM5YHTFNLMFVOJCCDKUVZQOZMMZJDM4YNFSNQQ, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:13.142160110 (597h3m13.013243145s) [noise=true , address=145.79.227.3:4151 , piece_id=OR3HCZMRMCCPRXF7BYSOBJGYGUJSVDD5XXXSFDRP6ZI46N6GQBGA, seg_pos={0 0}, upload=523359]: Uploading piece... Dec 30 12:32:14.856224368 (597h3m14.727307404s) [upload=523359]: Next returned nil segment Dec 30 12:32:14.856236812 (597h3m14.727319857s) [upload=523359]: Waiting for error group managing segments... Dec 30 12:32:15.130990464 (597h3m15.002073499s) [noise=true , address=145.79.227.46:4602 , piece_id=GDVL75E7RKKVTFLBXHG62ATXTTM47Q3FMY55T27QWJC6D45Y5BPA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.131687191 (597h3m15.002770236s) [noise=true , address=145.79.227.41:4087 , piece_id=44F3EC7ZJMSU67MYKRYCY5UQO6HY6F3F2SV5VYHVZGZDZVVMQSJQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.141779318 (597h3m15.012862353s) [noise=true , address=145.79.227.13:4804 , piece_id=RIR4H2QD43SFG7XM566BJQ3FDNPEQSDUK372P4DYVWYRG26NIP4A, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.151289698 (597h3m15.022372743s) [noise=true , address=145.79.227.14:4837 , piece_id=356H4GRGOSGRZQ5YATOJHA77CHNDRBQ5LH2HEZNFLPSQTMXHH2BA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.151843141 (597h3m15.022926177s) [noise=true , address=145.79.227.20:4438 , piece_id=PXTOH7Y7ABJX2ZBM5YHTFNLMFVOJCCDKUVZQOZMMZJDM4YNFSNQQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.155563436 (597h3m15.026646481s) [noise=true , address=145.79.227.3:4151 , piece_id=OR3HCZMRMCCPRXF7BYSOBJGYGUJSVDD5XXXSFDRP6ZI46N6GQBGA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.160190236 (597h3m15.031273281s) [noise=true , address=145.79.227.9:4365 , piece_id=CYX66KBGPQ6INE6Q4C5Y66IWJB6NRYNXAAKVSOOMSNT35AUDLIPA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.164178039 (597h3m15.035261084s) [noise=true , address=145.79.227.47:4267 , piece_id=R4ZTPFWP3XIIKOGJ7XVN6WZH2RRTZCA2WASDV33K36ZPOOBAZ7AQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.378427733 (597h3m15.249510768s) [noise=true , address=77.90.31.11:4554 , piece_id=3WRYHVO6L4B7QINWNUQ5QPCULKW3S2G5LWRBXFGSN2FI42PJFCUQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.421243280 (597h3m15.292326315s) [noise=true , address=77.90.31.17:4773 , piece_id=XFSUO4HI3NKNBBTTACKCD544IPVFZDGC7ITWELBJJMZC4LWT7CCA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.434260938 (597h3m15.305343973s) [noise=true , address=77.90.31.12:4590 , piece_id=WFHDEJEDWKCPXF64FDIWAZ35QHZAKYYX2JLIACE2NS3PQHOHLJZA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.446578473 (597h3m15.317661508s) [noise=true , address=45.91.116.8:4045 , piece_id=BKVMRLAOJGGPRUUE5VF2HXCIQ4JTQ555E36ISQFP7BUXI3UTULDA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.549935580 (597h3m15.421018615s) [noise=true , address=45.91.116.12:4226 , piece_id=4LKSOLZA42VW6CTPYPBNZMT5NYJ4XEZFZR7X4BILA2ZIQRWEYUIQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: <nil> Dec 30 12:32:15.549960006 (597h3m15.421043051s) [seg_pos={0 0}, upload=523359]: Segment reached optimal threshold of 13 pieces. Dec 30 12:32:15.550163925 (597h3m15.42124697s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34224->45.91.116.6:4976: use of closed network connection Dec 30 12:32:15.550174265 (597h3m15.4212573s) [noise=true , address=45.91.116.6:4976 , piece_id=2KTCLNNF5JSAETIN4PPUWMD3NU5VBILLSIIELHNMM2N7HKUHMJAQ, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550184334 (597h3m15.421267379s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.9:4264: use of closed network connection Dec 30 12:32:15.550190425 (597h3m15.42127347s) [noise=true , address=45.91.116.9:4264 , piece_id=IXYNP25XMTWAUO7GZXN7BDGJNV7D6QBPDLDYG3ULS62Q3OZHLFHA, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550484224 (597h3m15.421567259s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55516->45.91.116.10:4083: use of closed network connection Dec 30 12:32:15.550493433 (597h3m15.421576478s) [noise=true , address=45.91.116.10:4083 , piece_id=45BTTBV3Y2RO64WBBUI274H6MCQLVD5EGSJJUNPCGUZHBA2QFFDA, seg_pos={0 0}, upload=523359]: Long tail cancellation due to optimal threshold Dec 30 12:32:15.550516155 (597h3m15.42159919s) [seg_pos={0 0}, upload=523359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:32:15.550669267 (597h3m15.421752312s) [upload=523359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:32:15.550677433 (597h3m15.421760468s) [upload=523359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:32:15.845600259 (597h3m15.716683304s) [upload=523359]: Done uploading segments... Dec 30 12:32:15.845624025 (597h3m15.71670706s) [upload=523359]: Upload finished. err: <nil> Duration: 30.002341547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XXXX] duration: 5.001366004s trace: https://jaeger.dev.storj.io/trace/216b2ed9c632f71c FAIL (2025-12-30 12:32:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001380543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.................X..................X.....X.X...X.] duration: 41.616998594s trace: https://jaeger.dev.storj.io/trace/12ab50cae54c34b8 FAIL (2025-12-30 12:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3335]: Uploading piece... Dec 30 12:30:35.776704210 (597h1m35.647787255s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777174075 (597h1m35.64825712s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777679076 (597h1m35.648762121s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778158129 (597h1m35.649241164s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778618795 (597h1m35.649701841s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779090805 (597h1m35.65017385s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779577592 (597h1m35.650660637s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.780567807 (597h1m35.651650842s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.781941372 (597h1m35.653024417s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.783206401 (597h1m35.654289436s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.784920044 (597h1m35.656003089s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.786933338 (597h1m35.658016383s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.788728716 (597h1m35.659811751s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.790109045 (597h1m35.66119208s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.791393441 (597h1m35.662476486s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.792676213 (597h1m35.663759258s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:37.211774620 (597h1m37.082857655s) [upload=523335]: Next returned nil segment Dec 30 12:30:37.211791042 (597h1m37.082874077s) [upload=523335]: Waiting for error group managing segments... Dec 30 12:30:37.571833998 (597h1m37.442917033s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632153686 (597h1m37.503236721s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632641054 (597h1m37.503724089s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.645908035 (597h1m37.516991081s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.655521531 (597h1m37.526604576s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.719473996 (597h1m37.590557042s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.897438869 (597h1m37.768521904s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.945885809 (597h1m37.816968854s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.988490013 (597h1m37.859573048s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.499362532 (597h1m38.370445567s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.542379102 (597h1m38.413462138s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.568219716 (597h1m38.439302761s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.648331221 (597h1m38.519414256s) [noise=true , address=145.79.227.35:4512 , piece_id=6SWJYOSFS3QLAP3LFZV5G66EAINO5AXBWUZBMNZENWIHWVSCUCEA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659794188 (597h1m38.530877233s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659818734 (597h1m38.530901769s) [seg_pos={0 0}, upload=523335]: Segment reached optimal threshold of 14 pieces. Dec 30 12:30:38.660086344 (597h1m38.531169389s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660094008 (597h1m38.531177043s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660405782 (597h1m38.531488827s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection Dec 30 12:30:38.660413647 (597h1m38.531496702s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660430640 (597h1m38.531513675s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): internal: context canceled Dec 30 12:30:38.660470045 (597h1m38.53155308s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660561720 (597h1m38.531644765s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660569023 (597h1m38.531652068s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660606895 (597h1m38.53168994s) [seg_pos={0 0}, upload=523335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:30:38.660701275 (597h1m38.53178432s) [upload=523335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:38.660707267 (597h1m38.531790312s) [upload=523335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:38.952034268 (597h1m38.823117323s) [upload=523335]: Done uploading segments... Dec 30 12:30:38.952059425 (597h1m38.82314247s) [upload=523335]: Upload finished. err: <nil> Duration: 41.617013183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XXX] duration: 5.000854992s trace: https://jaeger.dev.storj.io/trace/d59b3fc7eaca9d4 FAIL (2025-12-30 12:31:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000877505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X..] duration: 5.872800813s trace: https://jaeger.dev.storj.io/trace/2b84c166f34d94fb FAIL (2025-12-30 12:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA) context deadline exceeded ======LOGS====== (597h1m22.521290829s) [upload=523332]: Uploading segments... Dec 30 12:30:22.650627423 (597h1m22.521710468s) [upload=523332]: Got next segment. Inline: false Dec 30 12:30:22.650671516 (597h1m22.521754552s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:30:22.650677999 (597h1m22.521761034s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:30:22.898116760 (597h1m22.769199805s) [seg_pos={0 0}, upload=523332]: Begin upload segment... Dec 30 12:30:22.898172727 (597h1m22.769255772s) [seg_pos={0 0}, upload=523332]: Done begin upload segment. Dec 30 12:30:22.898211881 (597h1m22.769294936s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898263669 (597h1m22.769346704s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898967520 (597h1m22.770050555s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899321454 (597h1m22.770404499s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899639780 (597h1m22.770722815s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899934651 (597h1m22.771017686s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900264650 (597h1m22.771347695s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900562747 (597h1m22.771645792s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900842239 (597h1m22.771925284s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.901231852 (597h1m22.772314897s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.902556013 (597h1m22.773639048s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904017857 (597h1m22.775100892s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904505035 (597h1m22.77558808s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904928060 (597h1m22.776011105s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905452528 (597h1m22.776535564s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905965355 (597h1m22.77704839s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:24.603303213 (597h1m24.474386258s) [upload=523332]: Next returned nil segment Dec 30 12:30:24.603332268 (597h1m24.474415313s) [upload=523332]: Waiting for error group managing segments... Dec 30 12:30:24.889792742 (597h1m24.760875777s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.890142298 (597h1m24.761225343s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.891264104 (597h1m24.762347139s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.892445884 (597h1m24.763528919s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.907204737 (597h1m24.778287773s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.908566841 (597h1m24.779649876s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.919102101 (597h1m24.790185146s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.922323085 (597h1m24.79340612s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.977269571 (597h1m24.848352606s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.207429110 (597h1m25.078512145s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.212992784 (597h1m25.08407584s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.232934476 (597h1m25.104017521s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257957574 (597h1m25.12904061s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257975378 (597h1m25.129058413s) [seg_pos={0 0}, upload=523332]: Segment reached optimal threshold of 13 pieces. Dec 30 12:30:25.258170149 (597h1m25.129253194s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:25.258176221 (597h1m25.129259266s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258310717 (597h1m25.129393752s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection Dec 30 12:30:25.258320094 (597h1m25.129403129s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258371262 (597h1m25.129454307s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection Dec 30 12:30:25.258377704 (597h1m25.129460749s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258393935 (597h1m25.12947697s) [seg_pos={0 0}, upload=523332]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:30:25.258460652 (597h1m25.129543697s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:25.258467214 (597h1m25.129550259s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:25.557975434 (597h1m25.429058469s) [upload=523332]: Done uploading segments... Dec 30 12:30:25.557999539 (597h1m25.429082584s) [upload=523332]: Upload finished. err: <nil> Duration: 5.872830178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.................X..................X.....X.X...X] duration: 1m0.004821629s trace: https://jaeger.dev.storj.io/trace/12ab50cae54c34b8 FAIL (2025-12-30 12:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3335]: Uploading piece... Dec 30 12:30:35.776704210 (597h1m35.647787255s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777174075 (597h1m35.64825712s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.777679076 (597h1m35.648762121s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778158129 (597h1m35.649241164s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.778618795 (597h1m35.649701841s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779090805 (597h1m35.65017385s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.779577592 (597h1m35.650660637s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.780567807 (597h1m35.651650842s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.781941372 (597h1m35.653024417s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.783206401 (597h1m35.654289436s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.784920044 (597h1m35.656003089s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.786933338 (597h1m35.658016383s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.788728716 (597h1m35.659811751s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.790109045 (597h1m35.66119208s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.791393441 (597h1m35.662476486s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:35.792676213 (597h1m35.663759258s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Uploading piece... Dec 30 12:30:37.211774620 (597h1m37.082857655s) [upload=523335]: Next returned nil segment Dec 30 12:30:37.211791042 (597h1m37.082874077s) [upload=523335]: Waiting for error group managing segments... Dec 30 12:30:37.571833998 (597h1m37.442917033s) [noise=true , address=64.49.13.137:4494 , piece_id=LPHVXLK7EHVTEMXCLAJ4XVE66TAFJF4CCTIP5VVCWLV57PJUSLGA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632153686 (597h1m37.503236721s) [noise=true , address=145.79.227.46:4605 , piece_id=VPOCPXYGSPG5QD22DQUWQEC5HRBLDZAWJTHT5J2Y653G6XDFBX5A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.632641054 (597h1m37.503724089s) [noise=true , address=145.79.227.3:4175 , piece_id=WTBUOAAJMKJVY2XZYPWNAUBSXVMBHYHFGKMSH3JLUIITHS5V5YAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.645908035 (597h1m37.516991081s) [noise=true , address=145.79.227.17:4991 , piece_id=2DQJ4W66W5POUMISLR5HRM4OQRXRFQQITBQ36S6R54GEUI6M7M5Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.655521531 (597h1m37.526604576s) [noise=true , address=64.49.13.26:4949 , piece_id=UYCTFO4BQXLF7IBHVN2SAJRTQPKYXE5JWK4XRX5DKO5CM5637LVA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.719473996 (597h1m37.590557042s) [noise=true , address=145.79.227.29:4956 , piece_id=WCXHKOH3VEUSKG5P64BQRRXMKP4OO6PB2S5CXV6VVEUKC6Z56IRA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.897438869 (597h1m37.768521904s) [noise=true , address=77.90.31.12:4599 , piece_id=2F6IMXHOQRNKBUKL6TQALRT7C5O4GPMWZTAGEPB5NXDBNJE2UI3Q, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.945885809 (597h1m37.816968854s) [noise=true , address=45.91.116.12:4233 , piece_id=R73CVQK3P7IE4CD2XNU34OVIP76FAK256W4RIKSFO4JCZJWNJ3ZA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:37.988490013 (597h1m37.859573048s) [noise=true , address=77.90.31.17:4788 , piece_id=POUWV4T2RWNRCXIETU6ENANNDVKDVDGVZTECR2QIBKN4IVTTJREQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.499362532 (597h1m38.370445567s) [noise=true , address=145.79.227.43:4165 , piece_id=AX37JRU2PYZYEAUC6F3ZMAPTGPMCT3WOT4FSP5KEXOCIIPT35OKA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.542379102 (597h1m38.413462138s) [noise=true , address=145.79.227.33:4874 , piece_id=AAAEYEYLLH4SHOPMLYIG7UHXA4Y6SL6IFK2NJBTCUX6KPP2RZKLQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.568219716 (597h1m38.439302761s) [noise=true , address=145.79.227.30:4766 , piece_id=MTWL724CA5RFCOLR6P3BWQ2GOOEEBAKBM5DR2DS6PLYY5GYHKUAA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.648331221 (597h1m38.519414256s) [noise=true , address=145.79.227.35:4512 , piece_id=6SWJYOSFS3QLAP3LFZV5G66EAINO5AXBWUZBMNZENWIHWVSCUCEA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659794188 (597h1m38.530877233s) [noise=true , address=145.79.227.34:4919 , piece_id=TXOEGOEVOOOOGB2XJAJY5ZTFOKWK757P6SQ4OUMW5HIXXFNAV2IQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: <nil> Dec 30 12:30:38.659818734 (597h1m38.530901769s) [seg_pos={0 0}, upload=523335]: Segment reached optimal threshold of 14 pieces. Dec 30 12:30:38.660086344 (597h1m38.531169389s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660094008 (597h1m38.531177043s) [noise=true , address=145.79.227.49:4358 , piece_id=UAEJTA65H5JCJYBC6MMG3RCGVYKBXV6ORPR65ZESERSNMKSJ7A2A, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660405782 (597h1m38.531488827s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->64.49.13.169:4710: use of closed network connection Dec 30 12:30:38.660413647 (597h1m38.531496702s) [noise=true , address=64.49.13.169:4710 , piece_id=A7NVDYXRBWKHICD34T3HSMQRG4PAG7XCTVYZC2NDUERBLHRH2LEQ, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660430640 (597h1m38.531513675s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): internal: context canceled Dec 30 12:30:38.660470045 (597h1m38.53155308s) [noise=true , address=45.91.116.7:4039 , piece_id=Y6BLUZOEUMYNZHSRNRIFYLLTZZOCIKFY6E6HWHMHUQ46EQDTMVXA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660561720 (597h1m38.531644765s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:38.660569023 (597h1m38.531652068s) [noise=true , address=145.79.227.40:4053 , piece_id=N7J2SB3ZJ7LZPRPTL647666SHGKMIWTA7HRPRHOJCHZH52FG3LHA, seg_pos={0 0}, upload=523335]: Long tail cancellation due to optimal threshold Dec 30 12:30:38.660606895 (597h1m38.53168994s) [seg_pos={0 0}, upload=523335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:30:38.660701275 (597h1m38.53178432s) [upload=523335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:38.660707267 (597h1m38.531790312s) [upload=523335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:38.952034268 (597h1m38.823117323s) [upload=523335]: Done uploading segments... Dec 30 12:30:38.952059425 (597h1m38.82314247s) [upload=523335]: Upload finished. err: <nil> Duration: 1m0.004845164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X.] duration: 7.987519578s trace: https://jaeger.dev.storj.io/trace/2b84c166f34d94fb FAIL (2025-12-30 12:31:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA) context deadline exceeded ======LOGS====== (597h1m22.521290829s) [upload=523332]: Uploading segments... Dec 30 12:30:22.650627423 (597h1m22.521710468s) [upload=523332]: Got next segment. Inline: false Dec 30 12:30:22.650671516 (597h1m22.521754552s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:30:22.650677999 (597h1m22.521761034s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:30:22.898116760 (597h1m22.769199805s) [seg_pos={0 0}, upload=523332]: Begin upload segment... Dec 30 12:30:22.898172727 (597h1m22.769255772s) [seg_pos={0 0}, upload=523332]: Done begin upload segment. Dec 30 12:30:22.898211881 (597h1m22.769294936s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898263669 (597h1m22.769346704s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898967520 (597h1m22.770050555s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899321454 (597h1m22.770404499s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899639780 (597h1m22.770722815s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899934651 (597h1m22.771017686s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900264650 (597h1m22.771347695s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900562747 (597h1m22.771645792s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900842239 (597h1m22.771925284s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.901231852 (597h1m22.772314897s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.902556013 (597h1m22.773639048s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904017857 (597h1m22.775100892s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904505035 (597h1m22.77558808s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904928060 (597h1m22.776011105s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905452528 (597h1m22.776535564s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905965355 (597h1m22.77704839s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:24.603303213 (597h1m24.474386258s) [upload=523332]: Next returned nil segment Dec 30 12:30:24.603332268 (597h1m24.474415313s) [upload=523332]: Waiting for error group managing segments... Dec 30 12:30:24.889792742 (597h1m24.760875777s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.890142298 (597h1m24.761225343s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.891264104 (597h1m24.762347139s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.892445884 (597h1m24.763528919s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.907204737 (597h1m24.778287773s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.908566841 (597h1m24.779649876s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.919102101 (597h1m24.790185146s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.922323085 (597h1m24.79340612s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.977269571 (597h1m24.848352606s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.207429110 (597h1m25.078512145s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.212992784 (597h1m25.08407584s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.232934476 (597h1m25.104017521s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257957574 (597h1m25.12904061s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257975378 (597h1m25.129058413s) [seg_pos={0 0}, upload=523332]: Segment reached optimal threshold of 13 pieces. Dec 30 12:30:25.258170149 (597h1m25.129253194s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:25.258176221 (597h1m25.129259266s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258310717 (597h1m25.129393752s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection Dec 30 12:30:25.258320094 (597h1m25.129403129s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258371262 (597h1m25.129454307s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection Dec 30 12:30:25.258377704 (597h1m25.129460749s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258393935 (597h1m25.12947697s) [seg_pos={0 0}, upload=523332]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:30:25.258460652 (597h1m25.129543697s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:25.258467214 (597h1m25.129550259s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:25.557975434 (597h1m25.429058469s) [upload=523332]: Done uploading segments... Dec 30 12:30:25.557999539 (597h1m25.429082584s) [upload=523332]: Upload finished. err: <nil> Duration: 7.987538764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..X] duration: 30.003298823s trace: https://jaeger.dev.storj.io/trace/2b84c166f34d94fb FAIL (2025-12-30 12:30:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA) context deadline exceeded ======LOGS====== (597h1m22.521290829s) [upload=523332]: Uploading segments... Dec 30 12:30:22.650627423 (597h1m22.521710468s) [upload=523332]: Got next segment. Inline: false Dec 30 12:30:22.650671516 (597h1m22.521754552s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:30:22.650677999 (597h1m22.521761034s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:30:22.898116760 (597h1m22.769199805s) [seg_pos={0 0}, upload=523332]: Begin upload segment... Dec 30 12:30:22.898172727 (597h1m22.769255772s) [seg_pos={0 0}, upload=523332]: Done begin upload segment. Dec 30 12:30:22.898211881 (597h1m22.769294936s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898263669 (597h1m22.769346704s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.898967520 (597h1m22.770050555s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899321454 (597h1m22.770404499s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899639780 (597h1m22.770722815s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.899934651 (597h1m22.771017686s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900264650 (597h1m22.771347695s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900562747 (597h1m22.771645792s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.900842239 (597h1m22.771925284s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.901231852 (597h1m22.772314897s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.902556013 (597h1m22.773639048s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904017857 (597h1m22.775100892s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904505035 (597h1m22.77558808s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.904928060 (597h1m22.776011105s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905452528 (597h1m22.776535564s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:22.905965355 (597h1m22.77704839s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Uploading piece... Dec 30 12:30:24.603303213 (597h1m24.474386258s) [upload=523332]: Next returned nil segment Dec 30 12:30:24.603332268 (597h1m24.474415313s) [upload=523332]: Waiting for error group managing segments... Dec 30 12:30:24.889792742 (597h1m24.760875777s) [noise=true , address=145.79.227.22:4510 , piece_id=P4AZATKYY3BFVLCPX52EW73TYTOLP6J7SY2XZFTQULAPJTTKZ35A, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.890142298 (597h1m24.761225343s) [noise=true , address=145.79.227.41:4085 , piece_id=H3BL2ML2YKV67RJ2OUQLG3Y3PJZ2ISYW3ABD2TPGVWY7L3TNCFBQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.891264104 (597h1m24.762347139s) [noise=true , address=145.79.227.51:4410 , piece_id=Q6W6FHBNUOWMPA3A3MPDQA7GODAU4Q3YNGG2FBXJOTLYNYPYPHCQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.892445884 (597h1m24.763528919s) [noise=true , address=145.79.227.15:4876 , piece_id=64M4RI64X2N3P7XN3VUFIT5BDJUJC6IGG6SK2ZYRIYSWWIXZMEKA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.907204737 (597h1m24.778287773s) [noise=true , address=145.79.227.52:4447 , piece_id=EIMLF5CQNFGOTNX57WWZYMIWC7VT45YC3WPK52JNJQXVWHFX56HQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.908566841 (597h1m24.779649876s) [noise=true , address=145.79.227.38:4978 , piece_id=U5SWIOG5WSYLYZDGIIDB2QCHHS7L55DHOVHAB4YPHRHXHV3Z5CVQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.919102101 (597h1m24.790185146s) [noise=true , address=145.79.227.42:4122 , piece_id=IH6WEBZB432LXRYFZLAS74GNE3YI73FUM4MUDXGH3KZFOTCP5TWA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.922323085 (597h1m24.79340612s) [noise=true , address=145.79.227.25:4619 , piece_id=6BBVLM6IER7GWKVUJPB27BXL6OZAZTGTWNJBIIFRT2PIJZ6J2D4Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:24.977269571 (597h1m24.848352606s) [noise=true , address=145.79.227.37:4566 , piece_id=QG75XPRZ3F2477DUE3RFAUJCTRFDRM7GWFECKRGM3I3VT42BXO3Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.207429110 (597h1m25.078512145s) [noise=true , address=45.91.116.13:4156 , piece_id=YOOWTQWK57WEJSVXX2YHURWIEZY5OVKEQ34ND2D7YS56A27SWXQQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.212992784 (597h1m25.08407584s) [noise=true , address=45.91.116.11:4119 , piece_id=PKWPJ25HW2MWWM3DKIW2I6VQZJ4IVWSPH637IDXAFP5QVKGQEJFQ, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.232934476 (597h1m25.104017521s) [noise=true , address=77.90.31.16:4736 , piece_id=5726RBFXBUKTXK5R4DSDWBGQ3CD4FGJ4YZ75WKRU55HUGCDCKOSA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257957574 (597h1m25.12904061s) [noise=true , address=45.91.116.9:4263 , piece_id=UKACQWXUBENZLLOR7Z6GCES4VENSLJ2BHFYOTWY4XXKEI2QQBTEA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: <nil> Dec 30 12:30:25.257975378 (597h1m25.129058413s) [seg_pos={0 0}, upload=523332]: Segment reached optimal threshold of 13 pieces. Dec 30 12:30:25.258170149 (597h1m25.129253194s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 12:30:25.258176221 (597h1m25.129259266s) [noise=true , address=145.79.227.40:4050 , piece_id=UGQN5TRACVL5RJ2S6GSPC3E6PCCIG3NWEWFPQSNNHJIMQ27REIRA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258310717 (597h1m25.129393752s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38044->45.91.116.8:4048: use of closed network connection Dec 30 12:30:25.258320094 (597h1m25.129403129s) [noise=true , address=45.91.116.8:4048 , piece_id=EBNDK4IAKCCE5JB3KOMGSRSEWJGUIYC4O45EYLCRNOYGPXLI3RYA, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258371262 (597h1m25.129454307s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37016->77.90.31.21:4877: use of closed network connection Dec 30 12:30:25.258377704 (597h1m25.129460749s) [noise=true , address=77.90.31.21:4877 , piece_id=B5ZBA4GJLFNTKR45B2VODC5IC26QTMSNHVJXGKTEQN4JWQV7B56Q, seg_pos={0 0}, upload=523332]: Long tail cancellation due to optimal threshold Dec 30 12:30:25.258393935 (597h1m25.12947697s) [seg_pos={0 0}, upload=523332]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:30:25.258460652 (597h1m25.129543697s) [upload=523332]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:30:25.258467214 (597h1m25.129550259s) [upload=523332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:30:25.557975434 (597h1m25.429058469s) [upload=523332]: Done uploading segments... Dec 30 12:30:25.557999539 (597h1m25.429082584s) [upload=523332]: Upload finished. err: <nil> Duration: 30.003323891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.XX] duration: 5.001251229s trace: https://jaeger.dev.storj.io/trace/1d6860ae746bb00 FAIL (2025-12-30 12:30:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001273743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.................X..................X.....X.X...] duration: 35.811934757s trace: https://jaeger.dev.storj.io/trace/2aec2f4396bc1d72 FAIL (2025-12-30 12:30:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.481758110 (596h57m16.352841155s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482205061 (596h57m16.353288096s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482605914 (596h57m16.353688959s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482859748 (596h57m16.353942793s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482991729 (596h57m16.354074764s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483379137 (596h57m16.354462182s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483827210 (596h57m16.354910255s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484233965 (596h57m16.35531702s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484652131 (596h57m16.355735176s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485067542 (596h57m16.356150587s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485223209 (596h57m16.356306244s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485872374 (596h57m16.356955419s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486478208 (596h57m16.357561244s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486887878 (596h57m16.357970923s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.487382861 (596h57m16.358465906s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:17.876958643 (596h57m17.748041688s) [upload=523272]: Next returned nil segment Dec 30 12:26:17.876971859 (596h57m17.748054894s) [upload=523272]: Waiting for error group managing segments... Dec 30 12:26:18.338080519 (596h57m18.209163555s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.340676402 (596h57m18.211759557s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.440672280 (596h57m18.311755315s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.502418053 (596h57m18.373501088s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.650954629 (596h57m18.522037664s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.665892864 (596h57m18.536975899s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.729711073 (596h57m18.600794108s) [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.790936746 (596h57m18.662019781s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.116992927 (596h57m18.988075972s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.232954315 (596h57m19.10403735s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.255520825 (596h57m19.12660386s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.270786533 (596h57m19.141869578s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.319842443 (596h57m19.190925488s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326809018 (596h57m19.197892063s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326829036 (596h57m19.197912071s) [seg_pos={0 0}, upload=523272]: Segment reached optimal threshold of 14 pieces. Dec 30 12:26:19.327072179 (596h57m19.198155224s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327082569 (596h57m19.198165614s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327506146 (596h57m19.198589181s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection Dec 30 12:26:19.327516645 (596h57m19.19859969s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327679496 (596h57m19.198762541s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327685557 (596h57m19.198768602s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327772192 (596h57m19.198855237s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327777402 (596h57m19.198860437s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327791799 (596h57m19.198874844s) [seg_pos={0 0}, upload=523272]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:26:19.327900586 (596h57m19.198983621s) [upload=523272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:26:19.327906518 (596h57m19.198989563s) [upload=523272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:26:19.639334514 (596h57m19.510417579s) [upload=523272]: Done uploading segments... Dec 30 12:26:19.639360604 (596h57m19.510443649s) [upload=523272]: Upload finished. err: <nil> Duration: 35.811964122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX..] duration: 5.737939054s trace: https://jaeger.dev.storj.io/trace/5a4d692592bd3d0a FAIL (2025-12-30 12:29:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ) context deadline exceeded ======LOGS====== Dec 30 12:28:43.091225840 (596h59m42.962308876s) [upload=523307]: Starting upload Dec 30 12:28:43.091308187 (596h59m42.962391222s) [upload=523307]: Done starting upload Dec 30 12:28:43.091378491 (596h59m42.962461526s) [upload=523307]: Uploading segments... Dec 30 12:28:43.091731212 (596h59m42.962814257s) [upload=523307]: Got next segment. Inline: false Dec 30 12:28:43.091787630 (596h59m42.962870675s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:28:43.091793241 (596h59m42.962876276s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:28:43.331396261 (596h59m43.202479296s) [seg_pos={0 0}, upload=523307]: Begin upload segment... Dec 30 12:28:43.331442119 (596h59m43.202525154s) [seg_pos={0 0}, upload=523307]: Done begin upload segment. Dec 30 12:28:43.331466014 (596h59m43.202549049s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.331495971 (596h59m43.202579006s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332280756 (596h59m43.203363791s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332562983 (596h59m43.203646018s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332825633 (596h59m43.203908668s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333096789 (596h59m43.204179824s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333598073 (596h59m43.204681109s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334023334 (596h59m43.205106369s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334490593 (596h59m43.205573638s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335496819 (596h59m43.206579865s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335616998 (596h59m43.206700033s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336757470 (596h59m43.207840515s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336937623 (596h59m43.208020658s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338050692 (596h59m43.209133727s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338277974 (596h59m43.209361009s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.339539978 (596h59m43.210623013s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:45.033332345 (596h59m44.90441538s) [upload=523307]: Next returned nil segment Dec 30 12:28:45.033350059 (596h59m44.904433094s) [upload=523307]: Waiting for error group managing segments... Dec 30 12:28:45.312762433 (596h59m45.183845478s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.325031836 (596h59m45.196114871s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.326438183 (596h59m45.197521218s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.330904989 (596h59m45.201988024s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.339473204 (596h59m45.210556239s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.343783241 (596h59m45.214866276s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.497748565 (596h59m45.3688316s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.631969622 (596h59m45.503052657s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.658535958 (596h59m45.529619003s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.690127092 (596h59m45.561210127s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.751129099 (596h59m45.622212134s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.854872863 (596h59m45.725955908s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355445409 (596h59m46.226528444s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355468122 (596h59m46.226551157s) [seg_pos={0 0}, upload=523307]: Segment reached optimal threshold of 13 pieces. Dec 30 12:28:46.355715022 (596h59m46.226798057s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355725632 (596h59m46.226808677s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355881229 (596h59m46.226964274s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355892220 (596h59m46.226975255s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355920794 (596h59m46.227003839s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355931574 (596h59m46.22701462s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355952695 (596h59m46.22703574s) [seg_pos={0 0}, upload=523307]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:28:46.356040882 (596h59m46.227123918s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:28:46.356048998 (596h59m46.227132053s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:28:46.653138702 (596h59m46.524221737s) [upload=523307]: Done uploading segments... Dec 30 12:28:46.653160724 (596h59m46.524243769s) [upload=523307]: Upload finished. err: <nil> Duration: 5.737962357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.X] duration: 5.001474203s trace: https://jaeger.dev.storj.io/trace/11dab46fcc383cfd FAIL (2025-12-30 12:29:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001517356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.................X..................X.....X.X..] duration: 45.381517104s trace: https://jaeger.dev.storj.io/trace/2aec2f4396bc1d72 FAIL (2025-12-30 12:29:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.481758110 (596h57m16.352841155s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482205061 (596h57m16.353288096s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482605914 (596h57m16.353688959s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482859748 (596h57m16.353942793s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482991729 (596h57m16.354074764s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483379137 (596h57m16.354462182s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483827210 (596h57m16.354910255s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484233965 (596h57m16.35531702s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484652131 (596h57m16.355735176s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485067542 (596h57m16.356150587s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485223209 (596h57m16.356306244s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485872374 (596h57m16.356955419s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486478208 (596h57m16.357561244s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486887878 (596h57m16.357970923s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.487382861 (596h57m16.358465906s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:17.876958643 (596h57m17.748041688s) [upload=523272]: Next returned nil segment Dec 30 12:26:17.876971859 (596h57m17.748054894s) [upload=523272]: Waiting for error group managing segments... Dec 30 12:26:18.338080519 (596h57m18.209163555s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.340676402 (596h57m18.211759557s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.440672280 (596h57m18.311755315s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.502418053 (596h57m18.373501088s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.650954629 (596h57m18.522037664s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.665892864 (596h57m18.536975899s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.729711073 (596h57m18.600794108s) [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.790936746 (596h57m18.662019781s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.116992927 (596h57m18.988075972s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.232954315 (596h57m19.10403735s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.255520825 (596h57m19.12660386s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.270786533 (596h57m19.141869578s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.319842443 (596h57m19.190925488s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326809018 (596h57m19.197892063s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326829036 (596h57m19.197912071s) [seg_pos={0 0}, upload=523272]: Segment reached optimal threshold of 14 pieces. Dec 30 12:26:19.327072179 (596h57m19.198155224s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327082569 (596h57m19.198165614s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327506146 (596h57m19.198589181s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection Dec 30 12:26:19.327516645 (596h57m19.19859969s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327679496 (596h57m19.198762541s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327685557 (596h57m19.198768602s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327772192 (596h57m19.198855237s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327777402 (596h57m19.198860437s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327791799 (596h57m19.198874844s) [seg_pos={0 0}, upload=523272]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:26:19.327900586 (596h57m19.198983621s) [upload=523272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:26:19.327906518 (596h57m19.198989563s) [upload=523272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:26:19.639334514 (596h57m19.510417579s) [upload=523272]: Done uploading segments... Dec 30 12:26:19.639360604 (596h57m19.510443649s) [upload=523272]: Upload finished. err: <nil> Duration: 45.381534066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX.] duration: 5.855559613s trace: https://jaeger.dev.storj.io/trace/5a4d692592bd3d0a FAIL (2025-12-30 12:29:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ) context deadline exceeded ======LOGS====== Dec 30 12:28:43.091225840 (596h59m42.962308876s) [upload=523307]: Starting upload Dec 30 12:28:43.091308187 (596h59m42.962391222s) [upload=523307]: Done starting upload Dec 30 12:28:43.091378491 (596h59m42.962461526s) [upload=523307]: Uploading segments... Dec 30 12:28:43.091731212 (596h59m42.962814257s) [upload=523307]: Got next segment. Inline: false Dec 30 12:28:43.091787630 (596h59m42.962870675s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:28:43.091793241 (596h59m42.962876276s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:28:43.331396261 (596h59m43.202479296s) [seg_pos={0 0}, upload=523307]: Begin upload segment... Dec 30 12:28:43.331442119 (596h59m43.202525154s) [seg_pos={0 0}, upload=523307]: Done begin upload segment. Dec 30 12:28:43.331466014 (596h59m43.202549049s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.331495971 (596h59m43.202579006s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332280756 (596h59m43.203363791s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332562983 (596h59m43.203646018s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332825633 (596h59m43.203908668s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333096789 (596h59m43.204179824s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333598073 (596h59m43.204681109s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334023334 (596h59m43.205106369s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334490593 (596h59m43.205573638s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335496819 (596h59m43.206579865s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335616998 (596h59m43.206700033s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336757470 (596h59m43.207840515s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336937623 (596h59m43.208020658s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338050692 (596h59m43.209133727s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338277974 (596h59m43.209361009s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.339539978 (596h59m43.210623013s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:45.033332345 (596h59m44.90441538s) [upload=523307]: Next returned nil segment Dec 30 12:28:45.033350059 (596h59m44.904433094s) [upload=523307]: Waiting for error group managing segments... Dec 30 12:28:45.312762433 (596h59m45.183845478s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.325031836 (596h59m45.196114871s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.326438183 (596h59m45.197521218s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.330904989 (596h59m45.201988024s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.339473204 (596h59m45.210556239s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.343783241 (596h59m45.214866276s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.497748565 (596h59m45.3688316s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.631969622 (596h59m45.503052657s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.658535958 (596h59m45.529619003s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.690127092 (596h59m45.561210127s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.751129099 (596h59m45.622212134s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.854872863 (596h59m45.725955908s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355445409 (596h59m46.226528444s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355468122 (596h59m46.226551157s) [seg_pos={0 0}, upload=523307]: Segment reached optimal threshold of 13 pieces. Dec 30 12:28:46.355715022 (596h59m46.226798057s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355725632 (596h59m46.226808677s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355881229 (596h59m46.226964274s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355892220 (596h59m46.226975255s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355920794 (596h59m46.227003839s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355931574 (596h59m46.22701462s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355952695 (596h59m46.22703574s) [seg_pos={0 0}, upload=523307]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:28:46.356040882 (596h59m46.227123918s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:28:46.356048998 (596h59m46.227132053s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:28:46.653138702 (596h59m46.524221737s) [upload=523307]: Done uploading segments... Dec 30 12:28:46.653160724 (596h59m46.524243769s) [upload=523307]: Upload finished. err: <nil> Duration: 5.855587277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.XX] duration: 30.002893719s trace: https://jaeger.dev.storj.io/trace/5a4d692592bd3d0a FAIL (2025-12-30 12:29:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ) context deadline exceeded ======LOGS====== Dec 30 12:28:43.091225840 (596h59m42.962308876s) [upload=523307]: Starting upload Dec 30 12:28:43.091308187 (596h59m42.962391222s) [upload=523307]: Done starting upload Dec 30 12:28:43.091378491 (596h59m42.962461526s) [upload=523307]: Uploading segments... Dec 30 12:28:43.091731212 (596h59m42.962814257s) [upload=523307]: Got next segment. Inline: false Dec 30 12:28:43.091787630 (596h59m42.962870675s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:28:43.091793241 (596h59m42.962876276s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:28:43.331396261 (596h59m43.202479296s) [seg_pos={0 0}, upload=523307]: Begin upload segment... Dec 30 12:28:43.331442119 (596h59m43.202525154s) [seg_pos={0 0}, upload=523307]: Done begin upload segment. Dec 30 12:28:43.331466014 (596h59m43.202549049s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.331495971 (596h59m43.202579006s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332280756 (596h59m43.203363791s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332562983 (596h59m43.203646018s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.332825633 (596h59m43.203908668s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333096789 (596h59m43.204179824s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.333598073 (596h59m43.204681109s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334023334 (596h59m43.205106369s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.334490593 (596h59m43.205573638s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335496819 (596h59m43.206579865s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.335616998 (596h59m43.206700033s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336757470 (596h59m43.207840515s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.336937623 (596h59m43.208020658s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338050692 (596h59m43.209133727s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.338277974 (596h59m43.209361009s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:43.339539978 (596h59m43.210623013s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Uploading piece... Dec 30 12:28:45.033332345 (596h59m44.90441538s) [upload=523307]: Next returned nil segment Dec 30 12:28:45.033350059 (596h59m44.904433094s) [upload=523307]: Waiting for error group managing segments... Dec 30 12:28:45.312762433 (596h59m45.183845478s) [noise=true , address=145.79.227.51:4410 , piece_id=4JYKVVOOA4C3FVCWQOO5HUP7JHKJR6RFME5SBDIHY3WKCWLADPJQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.325031836 (596h59m45.196114871s) [noise=true , address=145.79.227.5:4223 , piece_id=J2WZLFSZ22IP4O2YCE5FKLCBLW4XTVCN6NXW42M5NKI6K3BESEHQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.326438183 (596h59m45.197521218s) [noise=true , address=145.79.227.23:4545 , piece_id=VGKFAQD3PPTUWJFQ2ZA5237PAPDJRIUL4IJPPSS5QN2BGRT3L5IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.330904989 (596h59m45.201988024s) [noise=true , address=145.79.227.29:4942 , piece_id=5OTMN4UBU4XD34JAAWO3GVDURITLRMZCQPI5QYILHEA5ACMJZ53Q, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.339473204 (596h59m45.210556239s) [noise=true , address=145.79.227.15:4873 , piece_id=UA6JDS72IZ3OTBKNLCRD3HT6BQYTKEXY343IIYGZRM6JZEZWZWAQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.343783241 (596h59m45.214866276s) [noise=true , address=145.79.227.25:4619 , piece_id=J3MYQEVUQX6R343TEQH6VVZCXOKHRMEWSRDSW7RTOI4GQXI52JZA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.497748565 (596h59m45.3688316s) [noise=true , address=77.90.31.18:4805 , piece_id=CB2K37FKLQV37TW63EDSSL6NMHR7BTBRYMDUQEDM2YUUV4FBD2LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.631969622 (596h59m45.503052657s) [noise=true , address=77.90.31.14:4664 , piece_id=Y2LMS37VMPNAVHHWLNNTQ4FQ4TQDA3G2HNEG4XROU2H3QNC2ZU6A, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.658535958 (596h59m45.529619003s) [noise=true , address=45.91.116.13:4153 , piece_id=BVQDQABPRSHAUCDHDREJEWO5FNNVXFUHAIYFSA6D7X7Q4N4YTVMQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.690127092 (596h59m45.561210127s) [noise=true , address=45.91.116.10:4081 , piece_id=CBUECIR4E2RWHOKCQ3EXD4Z3E63T3WOJRWCRQX6GZXWK2CKHN7IQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.751129099 (596h59m45.622212134s) [noise=true , address=77.90.31.11:4556 , piece_id=W7PTZ5AUSGQUT23JTQVZTKHUWSWEPBJQDCSZG4AO5ZYCZO6XOKPQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:45.854872863 (596h59m45.725955908s) [noise=true , address=45.91.116.6:4974 , piece_id=6RZOJM27BKBSJPOC3A4YZKSX7HIVEU64NOJGRTBZGNGXPYS2IACQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355445409 (596h59m46.226528444s) [noise=true , address=45.91.116.11:4119 , piece_id=7VCCN43G7E72N6ZBWJCQ2TMBWD7YI5A4SSREQL2P2PM432RJRYBQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: <nil> Dec 30 12:28:46.355468122 (596h59m46.226551157s) [seg_pos={0 0}, upload=523307]: Segment reached optimal threshold of 13 pieces. Dec 30 12:28:46.355715022 (596h59m46.226798057s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355725632 (596h59m46.226808677s) [noise=true , address=45.91.116.12:4225 , piece_id=QLHWFUXSLN7KSFTQ2JVKUJWRH22MB3MBJZ47CZKAREL6USP77FYQ, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355881229 (596h59m46.226964274s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355892220 (596h59m46.226975255s) [noise=true , address=145.79.227.34:4906 , piece_id=YN4KTKE3WPWO3E2GPRXCCGEMSAUDB4U6Z227JTTO2U7ILKEI5KOA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355920794 (596h59m46.227003839s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 12:28:46.355931574 (596h59m46.22701462s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4HKZFFTL452T4QNU2GHIFELSPVGGKJTFAXIYJPUUSPM3Q7PU5LA, seg_pos={0 0}, upload=523307]: Long tail cancellation due to optimal threshold Dec 30 12:28:46.355952695 (596h59m46.22703574s) [seg_pos={0 0}, upload=523307]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:28:46.356040882 (596h59m46.227123918s) [upload=523307]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:28:46.356048998 (596h59m46.227132053s) [upload=523307]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:28:46.653138702 (596h59m46.524221737s) [upload=523307]: Done uploading segments... Dec 30 12:28:46.653160724 (596h59m46.524243769s) [upload=523307]: Upload finished. err: <nil> Duration: 30.002914137s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X.] duration: 3.579870669s trace: https://jaeger.dev.storj.io/trace/6272728cd7766cfa FAIL (2025-12-30 12:28:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579893813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...X] duration: 5.001189082s trace: https://jaeger.dev.storj.io/trace/6272728cd7766cfa FAIL (2025-12-30 12:28:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001212216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.................X..................X.....X.X.] duration: 48.594903812s trace: https://jaeger.dev.storj.io/trace/2aec2f4396bc1d72 FAIL (2025-12-30 12:28:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.481758110 (596h57m16.352841155s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482205061 (596h57m16.353288096s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482605914 (596h57m16.353688959s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482859748 (596h57m16.353942793s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482991729 (596h57m16.354074764s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483379137 (596h57m16.354462182s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483827210 (596h57m16.354910255s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484233965 (596h57m16.35531702s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484652131 (596h57m16.355735176s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485067542 (596h57m16.356150587s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485223209 (596h57m16.356306244s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485872374 (596h57m16.356955419s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486478208 (596h57m16.357561244s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486887878 (596h57m16.357970923s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.487382861 (596h57m16.358465906s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:17.876958643 (596h57m17.748041688s) [upload=523272]: Next returned nil segment Dec 30 12:26:17.876971859 (596h57m17.748054894s) [upload=523272]: Waiting for error group managing segments... Dec 30 12:26:18.338080519 (596h57m18.209163555s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.340676402 (596h57m18.211759557s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.440672280 (596h57m18.311755315s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.502418053 (596h57m18.373501088s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.650954629 (596h57m18.522037664s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.665892864 (596h57m18.536975899s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.729711073 (596h57m18.600794108s) [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.790936746 (596h57m18.662019781s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.116992927 (596h57m18.988075972s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.232954315 (596h57m19.10403735s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.255520825 (596h57m19.12660386s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.270786533 (596h57m19.141869578s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.319842443 (596h57m19.190925488s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326809018 (596h57m19.197892063s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326829036 (596h57m19.197912071s) [seg_pos={0 0}, upload=523272]: Segment reached optimal threshold of 14 pieces. Dec 30 12:26:19.327072179 (596h57m19.198155224s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327082569 (596h57m19.198165614s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327506146 (596h57m19.198589181s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection Dec 30 12:26:19.327516645 (596h57m19.19859969s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327679496 (596h57m19.198762541s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327685557 (596h57m19.198768602s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327772192 (596h57m19.198855237s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327777402 (596h57m19.198860437s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327791799 (596h57m19.198874844s) [seg_pos={0 0}, upload=523272]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:26:19.327900586 (596h57m19.198983621s) [upload=523272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:26:19.327906518 (596h57m19.198989563s) [upload=523272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:26:19.639334514 (596h57m19.510417579s) [upload=523272]: Done uploading segments... Dec 30 12:26:19.639360604 (596h57m19.510443649s) [upload=523272]: Upload finished. err: <nil> Duration: 48.594934571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X...X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.X] duration: 30.002183668s trace: https://jaeger.dev.storj.io/trace/1eaa4b9a34d83dc2 FAIL (2025-12-30 12:28:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:27:36.121443605 (596h58m35.992526641s) [upload=523291]: Starting upload Dec 30 12:27:36.121498410 (596h58m35.992581445s) [upload=523291]: Done starting upload Dec 30 12:27:36.121768975 (596h58m35.99285202s) [upload=523291]: Uploading segments... Dec 30 12:27:36.121835111 (596h58m35.992918146s) [upload=523291]: Got next segment. Inline: false Dec 30 12:27:36.121873884 (596h58m35.992956919s) [upload=523291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:27:36.121878163 (596h58m35.992961198s) [upload=523291]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:27:36.364748815 (596h58m36.23583185s) [seg_pos={0 0}, upload=523291]: Begin upload segment... Dec 30 12:27:36.364794402 (596h58m36.235877437s) [seg_pos={0 0}, upload=523291]: Done begin upload segment. Dec 30 12:27:36.364823387 (596h58m36.235906422s) [noise=true , address=145.79.227.32:4834 , piece_id=RO2LAOJIWKGMRK2MTTQ4OB56VTL5WBMLVYJY22YRVBBXK2X6K36Q, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.365234200 (596h58m36.236317245s) [noise=true , address=145.79.227.7:4293 , piece_id=FIRFO4OFBTHZFKP4RMVIFGOPKSDUGEWSGX5GUJLO5I6BMRWTJXBA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.366432892 (596h58m36.237515927s) [noise=true , address=145.79.227.19:4078 , piece_id=2WWUV2PKORJMKWGBHY5A4MO54QG2NDNIDB242RDQXVGBSFH5YWGA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.366845968 (596h58m36.237929003s) [noise=true , address=145.79.227.15:4876 , piece_id=MDTGGEQMAUKQTTGZDFDGWVRHF75C425T3ZUAIRBYQMHFHGCCIUOQ, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.367142603 (596h58m36.238225638s) [noise=true , address=145.79.227.9:4366 , piece_id=Q5NTPTLY2D5Q5JFD3OXHZAHV4F3FZERBLBEFZNCASQW5RJZI5M4A, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.367420442 (596h58m36.238503487s) [noise=true , address=77.90.31.17:4773 , piece_id=CYJZXANSCTUFOMZMB4T65ZLNBW745NT4CGE5AOXFRZX2I3QRYOBA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.367824842 (596h58m36.238907877s) [noise=true , address=145.79.227.28:4726 , piece_id=LXEHDB7WS7VDNK5KT6GWNTFQEVHLGCA5N3XSC6YG7TASKJIGWF2Q, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.369026680 (596h58m36.240109715s) [noise=true , address=145.79.227.30:4762 , piece_id=6JI2MTKGHSZJB2JEO4SB7C6CACIT5SGMM3RAHY5Z7AA75JGT6PNQ, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.369182457 (596h58m36.240265492s) [noise=true , address=77.90.31.15:4697 , piece_id=6BYCHVJ3ZD7XBAMQDLPXC3T5YNNH3N5AAFHURDHQCEEJN6Q5FVOA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.370427659 (596h58m36.241510704s) [noise=true , address=145.79.227.51:4411 , piece_id=J7QPDJNNYQVV56GQDGJC7OACKO7FLHINEQP3ILJHDMRPJR5R55AQ, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.370510977 (596h58m36.241594012s) [noise=true , address=145.79.227.24:4581 , piece_id=UTWGZLSTTUUGVNAYH7GJG3KZB2UEEKX3B5TQ3JFBYXG2DR7UWEIA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.371635498 (596h58m36.242718533s) [noise=true , address=145.79.227.39:4013 , piece_id=SJMCDLNTBKXCUUTNDPNYYWVN67KHVQGP74PLNYCXYDSGWFVXOTNA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.372954209 (596h58m36.244037254s) [noise=true , address=145.79.227.50:4373 , piece_id=MTWFWV6ZGUGLL2VAYUB3DJUU7L3YVGE4FNY3NZ4Z3HS25DNLXFJA, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.373865494 (596h58m36.244948539s) [noise=true , address=145.79.227.18:4007 , piece_id=HB535APBCUWGOFN2QLNJQNCICZFFE6U7YOM66FCJ2D32LT42D66A, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.375934484 (596h58m36.247017529s) [noise=true , address=45.91.116.6:4973 , piece_id=YEZRG7DS5KX44QAA2LPMMHL34M2ISU5TNFOMOH763LCF4MN36Q3Q, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:36.383547030 (596h58m36.254630065s) [noise=true , address=145.79.227.12:4765 , piece_id=KGZ6SJB5UP667ALVVGPBD6GOAPZK3HQ77LGN4VJPFTRPTEHUJL4A, seg_pos={0 0}, upload=523291]: Uploading piece... Dec 30 12:27:38.074124194 (596h58m37.945207239s) [upload=523291]: Next returned nil segment Dec 30 12:27:38.074142008 (596h58m37.945225043s) [upload=523291]: Waiting for error group managing segments... Dec 30 12:27:38.352951464 (596h58m38.224034509s) [noise=true , address=145.79.227.18:4007 , piece_id=HB535APBCUWGOFN2QLNJQNCICZFFE6U7YOM66FCJ2D32LT42D66A, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.359939409 (596h58m38.231022444s) [noise=true , address=145.79.227.19:4078 , piece_id=2WWUV2PKORJMKWGBHY5A4MO54QG2NDNIDB242RDQXVGBSFH5YWGA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.370529596 (596h58m38.241612631s) [noise=true , address=145.79.227.28:4726 , piece_id=LXEHDB7WS7VDNK5KT6GWNTFQEVHLGCA5N3XSC6YG7TASKJIGWF2Q, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.375632993 (596h58m38.246716028s) [noise=true , address=145.79.227.15:4876 , piece_id=MDTGGEQMAUKQTTGZDFDGWVRHF75C425T3ZUAIRBYQMHFHGCCIUOQ, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.378816335 (596h58m38.24989937s) [noise=true , address=145.79.227.50:4373 , piece_id=MTWFWV6ZGUGLL2VAYUB3DJUU7L3YVGE4FNY3NZ4Z3HS25DNLXFJA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.380612214 (596h58m38.251695259s) [noise=true , address=145.79.227.32:4834 , piece_id=RO2LAOJIWKGMRK2MTTQ4OB56VTL5WBMLVYJY22YRVBBXK2X6K36Q, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.381187579 (596h58m38.252270614s) [noise=true , address=145.79.227.12:4765 , piece_id=KGZ6SJB5UP667ALVVGPBD6GOAPZK3HQ77LGN4VJPFTRPTEHUJL4A, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.383471679 (596h58m38.254554714s) [noise=true , address=145.79.227.39:4013 , piece_id=SJMCDLNTBKXCUUTNDPNYYWVN67KHVQGP74PLNYCXYDSGWFVXOTNA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.399638613 (596h58m38.270721648s) [noise=true , address=145.79.227.9:4366 , piece_id=Q5NTPTLY2D5Q5JFD3OXHZAHV4F3FZERBLBEFZNCASQW5RJZI5M4A, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.439883495 (596h58m38.31096654s) [noise=true , address=145.79.227.51:4411 , piece_id=J7QPDJNNYQVV56GQDGJC7OACKO7FLHINEQP3ILJHDMRPJR5R55AQ, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.468274646 (596h58m38.339357691s) [noise=true , address=145.79.227.24:4581 , piece_id=UTWGZLSTTUUGVNAYH7GJG3KZB2UEEKX3B5TQ3JFBYXG2DR7UWEIA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.557365820 (596h58m38.428448865s) [noise=true , address=77.90.31.15:4697 , piece_id=6BYCHVJ3ZD7XBAMQDLPXC3T5YNNH3N5AAFHURDHQCEEJN6Q5FVOA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.566533637 (596h58m38.437616682s) [noise=true , address=77.90.31.17:4773 , piece_id=CYJZXANSCTUFOMZMB4T65ZLNBW745NT4CGE5AOXFRZX2I3QRYOBA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: <nil> Dec 30 12:27:38.566560318 (596h58m38.437643363s) [seg_pos={0 0}, upload=523291]: Segment reached optimal threshold of 13 pieces. Dec 30 12:27:38.566789383 (596h58m38.437872428s) [noise=true , address=145.79.227.30:4762 , piece_id=6JI2MTKGHSZJB2JEO4SB7C6CACIT5SGMM3RAHY5Z7AA75JGT6PNQ, seg_pos={0 0}, upload=523291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 12:27:38.566799673 (596h58m38.437882709s) [noise=true , address=145.79.227.30:4762 , piece_id=6JI2MTKGHSZJB2JEO4SB7C6CACIT5SGMM3RAHY5Z7AA75JGT6PNQ, seg_pos={0 0}, upload=523291]: Long tail cancellation due to optimal threshold Dec 30 12:27:38.566927356 (596h58m38.438010401s) [noise=true , address=145.79.227.7:4293 , piece_id=FIRFO4OFBTHZFKP4RMVIFGOPKSDUGEWSGX5GUJLO5I6BMRWTJXBA, seg_pos={0 0}, upload=523291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 30 12:27:38.566948717 (596h58m38.438031752s) [noise=true , address=145.79.227.7:4293 , piece_id=FIRFO4OFBTHZFKP4RMVIFGOPKSDUGEWSGX5GUJLO5I6BMRWTJXBA, seg_pos={0 0}, upload=523291]: Long tail cancellation due to optimal threshold Dec 30 12:27:38.567113070 (596h58m38.438196115s) [noise=true , address=45.91.116.6:4973 , piece_id=YEZRG7DS5KX44QAA2LPMMHL34M2ISU5TNFOMOH763LCF4MN36Q3Q, seg_pos={0 0}, upload=523291]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 30 12:27:38.567123780 (596h58m38.438206825s) [noise=true , address=45.91.116.6:4973 , piece_id=YEZRG7DS5KX44QAA2LPMMHL34M2ISU5TNFOMOH763LCF4MN36Q3Q, seg_pos={0 0}, upload=523291]: Long tail cancellation due to optimal threshold Dec 30 12:27:38.567156663 (596h58m38.438239708s) [seg_pos={0 0}, upload=523291]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:27:38.567245512 (596h58m38.438328567s) [upload=523291]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:27:38.567253047 (596h58m38.438336092s) [upload=523291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:27:38.858090825 (596h58m38.72917387s) [upload=523291]: Done uploading segments... Dec 30 12:27:38.858110563 (596h58m38.729193608s) [upload=523291]: Upload finished. err: <nil> Duration: 30.002208826s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX...] duration: 3.752360423s trace: https://jaeger.dev.storj.io/trace/b1541968540caef FAIL (2025-12-30 12:28:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.752414817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX..] duration: 3.614086937s trace: https://jaeger.dev.storj.io/trace/b1541968540caef FAIL (2025-12-30 12:27:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.614126281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.................X..................X.....X.X] duration: 1m0.002610961s trace: https://jaeger.dev.storj.io/trace/2aec2f4396bc1d72 FAIL (2025-12-30 12:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.481758110 (596h57m16.352841155s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482205061 (596h57m16.353288096s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482605914 (596h57m16.353688959s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482859748 (596h57m16.353942793s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.482991729 (596h57m16.354074764s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483379137 (596h57m16.354462182s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.483827210 (596h57m16.354910255s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484233965 (596h57m16.35531702s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.484652131 (596h57m16.355735176s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485067542 (596h57m16.356150587s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485223209 (596h57m16.356306244s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.485872374 (596h57m16.356955419s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486478208 (596h57m16.357561244s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.486887878 (596h57m16.357970923s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:16.487382861 (596h57m16.358465906s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Uploading piece... Dec 30 12:26:17.876958643 (596h57m17.748041688s) [upload=523272]: Next returned nil segment Dec 30 12:26:17.876971859 (596h57m17.748054894s) [upload=523272]: Waiting for error group managing segments... Dec 30 12:26:18.338080519 (596h57m18.209163555s) [noise=true , address=145.79.227.5:4249 , piece_id=E3FXXA5WGA7OT45FHSIRPLQJBODM6YR5HLKKFGTRTOHOZQAW7KVQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.340676402 (596h57m18.211759557s) [noise=true , address=145.79.227.10:4405 , piece_id=FWYBXFAII3PKJ2TIXGFMJK6XYAQZWWLMSUKIZHZZK4UKKVRPHC7Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.440672280 (596h57m18.311755315s) [noise=true , address=145.79.227.35:4485 , piece_id=SPZ47KSS36AB2ZM66NT562HU54NPRKFPFHB22ZKP2Z5SR77NXB4Q, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.502418053 (596h57m18.373501088s) [noise=true , address=77.90.31.21:4911 , piece_id=TX56VHV5ALL6LHLNC3R36LP2ZWDTXGJOI2UW7CGLBCNCJFZAK5UA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.650954629 (596h57m18.522037664s) [noise=true , address=77.90.31.12:4593 , piece_id=D7GJREACFCDWQZEJSFDCD5FSBKAVHEXBT3H3ECWX3EAZUJB4NCAA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.665892864 (596h57m18.536975899s) [noise=true , address=77.90.31.13:4630 , piece_id=UXNSJFPMJ2YTT4NQL6O77I3V5XNKNSLQXO24JRAWOPOT4CZWHC6A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.729711073 (596h57m18.600794108s) [noise=true , address=45.91.116.11:4150 , piece_id=65HNF7RAP2YXPBGFKWWOPR4LQ2TDYJF7PNH2BVVYNKKZPBKQOMUQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:18.790936746 (596h57m18.662019781s) [noise=true , address=45.91.116.14:4222 , piece_id=H7Y3QUD52LJS2CY7ZDAJVOZH5XJDLT3YGCJLSLSTOTI7NRLJVI5A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.116992927 (596h57m18.988075972s) [noise=true , address=193.149.164.143:4701 , piece_id=6B7MI4CJIL6SAXKLQE7PLHR7RU2PYYJJOPGGVFO42PBW57BSEDAQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.232954315 (596h57m19.10403735s) [noise=true , address=145.79.227.40:4059 , piece_id=YRSGMBRUNTN56Y7NXODCA2GVYI54MQKNHT7KUO3QVWGQH35F3DLQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.255520825 (596h57m19.12660386s) [noise=true , address=145.79.227.30:4791 , piece_id=GCTL2GP6Y5YY7VHFSJO4RDPXQHKIA2IOL7Q3SQLW4QSPFCKKEZFA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.270786533 (596h57m19.141869578s) [noise=true , address=145.79.227.43:4161 , piece_id=5VQL4H7MUOCT25UVUZLEDVIU2J23NACDXREWRTHA53HO74Q3QQCQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.319842443 (596h57m19.190925488s) [noise=true , address=145.79.227.44:4200 , piece_id=JNK6CMVGN3W2WX3ZWHSM6I3CXKVLBXVNKVP7OEYKALE47WYI24EA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326809018 (596h57m19.197892063s) [noise=true , address=145.79.227.49:4348 , piece_id=MOUA76JAPF3RWM2ZZULV56ZHIX6M2ZWZ7NUHYL5O3N7CMYOHODHA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: <nil> Dec 30 12:26:19.326829036 (596h57m19.197912071s) [seg_pos={0 0}, upload=523272]: Segment reached optimal threshold of 14 pieces. Dec 30 12:26:19.327072179 (596h57m19.198155224s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327082569 (596h57m19.198165614s) [noise=true , address=145.79.227.29:4949 , piece_id=BIRQKKJ6DVQO6YOA6ZEPLQZ7MABED5A2T7VFBIUEY5PT4S6HPPIQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327506146 (596h57m19.198589181s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44810->64.49.13.169:4715: use of closed network connection Dec 30 12:26:19.327516645 (596h57m19.19859969s) [noise=true , address=64.49.13.169:4715 , piece_id=U4POMG2334GJJ5TFBPPR2KFI5ZWYYTUOBDN7QHACEKLRW5SEAFBA, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327679496 (596h57m19.198762541s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327685557 (596h57m19.198768602s) [noise=true , address=145.79.227.34:4937 , piece_id=Y34R7SMAVSXWNC2P4A5X5MBADKNUB7QCFSTS62O2LARXOPOEY6VQ, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327772192 (596h57m19.198855237s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Dec 30 12:26:19.327777402 (596h57m19.198860437s) [noise=true , address=77.90.31.19:4860 , piece_id=XMLUCQRHG3F7NAJGVFLMGOQDREIVV4OVEUQWCPCFU7ALROSHQY4A, seg_pos={0 0}, upload=523272]: Long tail cancellation due to optimal threshold Dec 30 12:26:19.327791799 (596h57m19.198874844s) [seg_pos={0 0}, upload=523272]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:26:19.327900586 (596h57m19.198983621s) [upload=523272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:26:19.327906518 (596h57m19.198989563s) [upload=523272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:26:19.639334514 (596h57m19.510417579s) [upload=523272]: Done uploading segments... Dec 30 12:26:19.639360604 (596h57m19.510443649s) [upload=523272]: Upload finished. err: <nil> Duration: 1m0.00262086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X..X..X.X....X.X.......XX......XX.XX....XX..X.......X.] duration: 5.684581083s trace: https://jaeger.dev.storj.io/trace/2eaff7c6e2de435e FAIL (2025-12-30 12:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:25:55.298727783 (596h56m55.169810828s) [upload=523267]: Starting upload Dec 30 12:25:55.298851559 (596h56m55.169934594s) [upload=523267]: Done starting upload Dec 30 12:25:55.299184824 (596h56m55.170267869s) [upload=523267]: Uploading segments... Dec 30 12:25:55.299928069 (596h56m55.171011114s) [upload=523267]: Got next segment. Inline: false Dec 30 12:25:55.299988003 (596h56m55.171071038s) [upload=523267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:25:55.299994315 (596h56m55.17107735s) [upload=523267]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:25:55.547241240 (596h56m55.418324275s) [seg_pos={0 0}, upload=523267]: Begin upload segment... Dec 30 12:25:55.547300654 (596h56m55.418383689s) [seg_pos={0 0}, upload=523267]: Done begin upload segment. Dec 30 12:25:55.547330330 (596h56m55.418413375s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.547373312 (596h56m55.418456347s) [noise=true , address=45.91.116.9:4262 , piece_id=5R5UQXMLZTL3GABDG4NVQF4SQGLPUQGL7GPB73AAZQZG5KMVW5FQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.547874918 (596h56m55.418957963s) [noise=true , address=77.90.31.19:4842 , piece_id=TWGXSP76EOJCBZUFIVL27AQPIYQOEZYTU5AQBPRLC54XDZM7YUAA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548244360 (596h56m55.419327396s) [noise=true , address=145.79.227.14:4837 , piece_id=WLDC53VW7S4WD6YCZWJSHQSNZAH5TDRNW7ZGQFJWORWTZEGVMSYQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548350192 (596h56m55.419433227s) [noise=true , address=45.91.116.10:4084 , piece_id=XYMU6VMWPLH7JGNQNJB65M5XF254VNWTZGEY5Z32DSMRUP2IKWHQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548547147 (596h56m55.419630182s) [noise=true , address=145.79.227.39:4015 , piece_id=XMXREMYOSRXHWXBZBJLHY6E6WFGFXKNRW46AILVZJMWTBRS73IAA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548678107 (596h56m55.419761142s) [noise=true , address=145.79.227.16:4909 , piece_id=G2QDP5VUQL2HMNRNXQHUI2O3G4RTYC3MCSOJXCNX5PF64WR4UIXA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548918645 (596h56m55.42000169s) [noise=true , address=145.79.227.36:4517 , piece_id=EILKLKUB22UAV7RSFTWNTXDQNJ4OOGGOPNHRJCDCRMJZQEHVVG7Q, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548937551 (596h56m55.420020586s) [noise=true , address=145.79.227.17:4971 , piece_id=4BRLCAKJHTJGVUEI5G7MM443YUEBLNH5MG32HJLGJHTW6EY52EGQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549218135 (596h56m55.42030119s) [noise=true , address=145.79.227.23:4545 , piece_id=XOPOIK67FANRT7TNLAZK3ER752XRR3F4UHH36KKMPB2Z4CI3FYZQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549227212 (596h56m55.420310247s) [noise=true , address=45.91.116.14:4191 , piece_id=LI26LQR4CWT4B3KZCYO5GOUEXN6ZL3GHIRSEFRHTF3OL5EZMJYYQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549609079 (596h56m55.420692114s) [noise=true , address=77.90.31.14:4664 , piece_id=HBM57WOEQJXU4IAYQNHL6WCC2M5IMMSSH3KEDZFR4U7NVMRVJVPA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549621944 (596h56m55.420704979s) [noise=true , address=145.79.227.13:4804 , piece_id=ONPWV6E3IFXZZ2324PAL252OROUOQGXZU2ITVF4MYBPF222NFQXQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550007328 (596h56m55.421090363s) [noise=true , address=45.91.116.7:4009 , piece_id=IMQZ3JXCCVTS7M73BX5UZEKQE4TDQGK3ILBAZV5UHYJWZD2XWFXA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550108581 (596h56m55.421191626s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550338558 (596h56m55.421421603s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:57.239319904 (596h56m57.110402939s) [upload=523267]: Next returned nil segment Dec 30 12:25:57.239336576 (596h56m57.110419612s) [upload=523267]: Waiting for error group managing segments... Dec 30 12:25:57.528938703 (596h56m57.400021738s) [noise=true , address=145.79.227.36:4517 , piece_id=EILKLKUB22UAV7RSFTWNTXDQNJ4OOGGOPNHRJCDCRMJZQEHVVG7Q, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.540595279 (596h56m57.411678324s) [noise=true , address=145.79.227.23:4545 , piece_id=XOPOIK67FANRT7TNLAZK3ER752XRR3F4UHH36KKMPB2Z4CI3FYZQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.556119779 (596h56m57.427202824s) [noise=true , address=145.79.227.16:4909 , piece_id=G2QDP5VUQL2HMNRNXQHUI2O3G4RTYC3MCSOJXCNX5PF64WR4UIXA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.558579824 (596h56m57.429662859s) [noise=true , address=145.79.227.39:4015 , piece_id=XMXREMYOSRXHWXBZBJLHY6E6WFGFXKNRW46AILVZJMWTBRS73IAA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.564777666 (596h56m57.435860711s) [noise=true , address=145.79.227.13:4804 , piece_id=ONPWV6E3IFXZZ2324PAL252OROUOQGXZU2ITVF4MYBPF222NFQXQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.609926396 (596h56m57.481009441s) [noise=true , address=145.79.227.14:4837 , piece_id=WLDC53VW7S4WD6YCZWJSHQSNZAH5TDRNW7ZGQFJWORWTZEGVMSYQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.620690442 (596h56m57.491773487s) [noise=true , address=145.79.227.17:4971 , piece_id=4BRLCAKJHTJGVUEI5G7MM443YUEBLNH5MG32HJLGJHTW6EY52EGQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.734925773 (596h56m57.606008808s) [noise=true , address=77.90.31.19:4842 , piece_id=TWGXSP76EOJCBZUFIVL27AQPIYQOEZYTU5AQBPRLC54XDZM7YUAA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.863896543 (596h56m57.734979588s) [noise=true , address=77.90.31.14:4664 , piece_id=HBM57WOEQJXU4IAYQNHL6WCC2M5IMMSSH3KEDZFR4U7NVMRVJVPA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.890581886 (596h56m57.761664921s) [noise=true , address=45.91.116.14:4191 , piece_id=LI26LQR4CWT4B3KZCYO5GOUEXN6ZL3GHIRSEFRHTF3OL5EZMJYYQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.891444208 (596h56m57.762527253s) [noise=true , address=45.91.116.9:4262 , piece_id=5R5UQXMLZTL3GABDG4NVQF4SQGLPUQGL7GPB73AAZQZG5KMVW5FQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.948575394 (596h56m57.81965843s) [noise=true , address=45.91.116.10:4084 , piece_id=XYMU6VMWPLH7JGNQNJB65M5XF254VNWTZGEY5Z32DSMRUP2IKWHQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.985641183 (596h56m57.856724218s) [noise=true , address=45.91.116.7:4009 , piece_id=IMQZ3JXCCVTS7M73BX5UZEKQE4TDQGK3ILBAZV5UHYJWZD2XWFXA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.985667484 (596h56m57.856750529s) [seg_pos={0 0}, upload=523267]: Segment reached optimal threshold of 13 pieces. Dec 30 12:25:57.986026988 (596h56m57.857110033s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986048799 (596h56m57.857131834s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986193896 (596h56m57.857276941s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986205047 (596h56m57.857288082s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986437690 (596h56m57.857520725s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986448060 (596h56m57.857531105s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986470513 (596h56m57.857553558s) [seg_pos={0 0}, upload=523267]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:25:57.986551988 (596h56m57.857635033s) [upload=523267]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:25:57.986559051 (596h56m57.857642096s) [upload=523267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:25:58.279694175 (596h56m58.15077722s) [upload=523267]: Done uploading segments... Dec 30 12:25:58.279734552 (596h56m58.150817597s) [upload=523267]: Upload finished. err: <nil> Duration: 5.684598236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X...X..X..X.X....X.X.......XX......XX.XX....XX..X.......X] duration: 30.003495676s trace: https://jaeger.dev.storj.io/trace/2eaff7c6e2de435e FAIL (2025-12-30 12:26:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:25:55.298727783 (596h56m55.169810828s) [upload=523267]: Starting upload Dec 30 12:25:55.298851559 (596h56m55.169934594s) [upload=523267]: Done starting upload Dec 30 12:25:55.299184824 (596h56m55.170267869s) [upload=523267]: Uploading segments... Dec 30 12:25:55.299928069 (596h56m55.171011114s) [upload=523267]: Got next segment. Inline: false Dec 30 12:25:55.299988003 (596h56m55.171071038s) [upload=523267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:25:55.299994315 (596h56m55.17107735s) [upload=523267]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:25:55.547241240 (596h56m55.418324275s) [seg_pos={0 0}, upload=523267]: Begin upload segment... Dec 30 12:25:55.547300654 (596h56m55.418383689s) [seg_pos={0 0}, upload=523267]: Done begin upload segment. Dec 30 12:25:55.547330330 (596h56m55.418413375s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.547373312 (596h56m55.418456347s) [noise=true , address=45.91.116.9:4262 , piece_id=5R5UQXMLZTL3GABDG4NVQF4SQGLPUQGL7GPB73AAZQZG5KMVW5FQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.547874918 (596h56m55.418957963s) [noise=true , address=77.90.31.19:4842 , piece_id=TWGXSP76EOJCBZUFIVL27AQPIYQOEZYTU5AQBPRLC54XDZM7YUAA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548244360 (596h56m55.419327396s) [noise=true , address=145.79.227.14:4837 , piece_id=WLDC53VW7S4WD6YCZWJSHQSNZAH5TDRNW7ZGQFJWORWTZEGVMSYQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548350192 (596h56m55.419433227s) [noise=true , address=45.91.116.10:4084 , piece_id=XYMU6VMWPLH7JGNQNJB65M5XF254VNWTZGEY5Z32DSMRUP2IKWHQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548547147 (596h56m55.419630182s) [noise=true , address=145.79.227.39:4015 , piece_id=XMXREMYOSRXHWXBZBJLHY6E6WFGFXKNRW46AILVZJMWTBRS73IAA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548678107 (596h56m55.419761142s) [noise=true , address=145.79.227.16:4909 , piece_id=G2QDP5VUQL2HMNRNXQHUI2O3G4RTYC3MCSOJXCNX5PF64WR4UIXA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548918645 (596h56m55.42000169s) [noise=true , address=145.79.227.36:4517 , piece_id=EILKLKUB22UAV7RSFTWNTXDQNJ4OOGGOPNHRJCDCRMJZQEHVVG7Q, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.548937551 (596h56m55.420020586s) [noise=true , address=145.79.227.17:4971 , piece_id=4BRLCAKJHTJGVUEI5G7MM443YUEBLNH5MG32HJLGJHTW6EY52EGQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549218135 (596h56m55.42030119s) [noise=true , address=145.79.227.23:4545 , piece_id=XOPOIK67FANRT7TNLAZK3ER752XRR3F4UHH36KKMPB2Z4CI3FYZQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549227212 (596h56m55.420310247s) [noise=true , address=45.91.116.14:4191 , piece_id=LI26LQR4CWT4B3KZCYO5GOUEXN6ZL3GHIRSEFRHTF3OL5EZMJYYQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549609079 (596h56m55.420692114s) [noise=true , address=77.90.31.14:4664 , piece_id=HBM57WOEQJXU4IAYQNHL6WCC2M5IMMSSH3KEDZFR4U7NVMRVJVPA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.549621944 (596h56m55.420704979s) [noise=true , address=145.79.227.13:4804 , piece_id=ONPWV6E3IFXZZ2324PAL252OROUOQGXZU2ITVF4MYBPF222NFQXQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550007328 (596h56m55.421090363s) [noise=true , address=45.91.116.7:4009 , piece_id=IMQZ3JXCCVTS7M73BX5UZEKQE4TDQGK3ILBAZV5UHYJWZD2XWFXA, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550108581 (596h56m55.421191626s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:55.550338558 (596h56m55.421421603s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Uploading piece... Dec 30 12:25:57.239319904 (596h56m57.110402939s) [upload=523267]: Next returned nil segment Dec 30 12:25:57.239336576 (596h56m57.110419612s) [upload=523267]: Waiting for error group managing segments... Dec 30 12:25:57.528938703 (596h56m57.400021738s) [noise=true , address=145.79.227.36:4517 , piece_id=EILKLKUB22UAV7RSFTWNTXDQNJ4OOGGOPNHRJCDCRMJZQEHVVG7Q, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.540595279 (596h56m57.411678324s) [noise=true , address=145.79.227.23:4545 , piece_id=XOPOIK67FANRT7TNLAZK3ER752XRR3F4UHH36KKMPB2Z4CI3FYZQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.556119779 (596h56m57.427202824s) [noise=true , address=145.79.227.16:4909 , piece_id=G2QDP5VUQL2HMNRNXQHUI2O3G4RTYC3MCSOJXCNX5PF64WR4UIXA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.558579824 (596h56m57.429662859s) [noise=true , address=145.79.227.39:4015 , piece_id=XMXREMYOSRXHWXBZBJLHY6E6WFGFXKNRW46AILVZJMWTBRS73IAA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.564777666 (596h56m57.435860711s) [noise=true , address=145.79.227.13:4804 , piece_id=ONPWV6E3IFXZZ2324PAL252OROUOQGXZU2ITVF4MYBPF222NFQXQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.609926396 (596h56m57.481009441s) [noise=true , address=145.79.227.14:4837 , piece_id=WLDC53VW7S4WD6YCZWJSHQSNZAH5TDRNW7ZGQFJWORWTZEGVMSYQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.620690442 (596h56m57.491773487s) [noise=true , address=145.79.227.17:4971 , piece_id=4BRLCAKJHTJGVUEI5G7MM443YUEBLNH5MG32HJLGJHTW6EY52EGQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.734925773 (596h56m57.606008808s) [noise=true , address=77.90.31.19:4842 , piece_id=TWGXSP76EOJCBZUFIVL27AQPIYQOEZYTU5AQBPRLC54XDZM7YUAA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.863896543 (596h56m57.734979588s) [noise=true , address=77.90.31.14:4664 , piece_id=HBM57WOEQJXU4IAYQNHL6WCC2M5IMMSSH3KEDZFR4U7NVMRVJVPA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.890581886 (596h56m57.761664921s) [noise=true , address=45.91.116.14:4191 , piece_id=LI26LQR4CWT4B3KZCYO5GOUEXN6ZL3GHIRSEFRHTF3OL5EZMJYYQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.891444208 (596h56m57.762527253s) [noise=true , address=45.91.116.9:4262 , piece_id=5R5UQXMLZTL3GABDG4NVQF4SQGLPUQGL7GPB73AAZQZG5KMVW5FQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.948575394 (596h56m57.81965843s) [noise=true , address=45.91.116.10:4084 , piece_id=XYMU6VMWPLH7JGNQNJB65M5XF254VNWTZGEY5Z32DSMRUP2IKWHQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.985641183 (596h56m57.856724218s) [noise=true , address=45.91.116.7:4009 , piece_id=IMQZ3JXCCVTS7M73BX5UZEKQE4TDQGK3ILBAZV5UHYJWZD2XWFXA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: <nil> Dec 30 12:25:57.985667484 (596h56m57.856750529s) [seg_pos={0 0}, upload=523267]: Segment reached optimal threshold of 13 pieces. Dec 30 12:25:57.986026988 (596h56m57.857110033s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986048799 (596h56m57.857131834s) [noise=true , address=77.90.31.10:4519 , piece_id=H7AGWRXWXYVFH7LU25FDIASZIY4UTEUZNY27CR23IIZMTIPVWKEA, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986193896 (596h56m57.857276941s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986205047 (596h56m57.857288082s) [noise=true , address=145.79.227.29:4942 , piece_id=6GP6GZRGDY6WBZEEQJA5GLIIN5IX2HAMH5AVUL2EIKQBXMNL53SQ, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986437690 (596h56m57.857520725s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 12:25:57.986448060 (596h56m57.857531105s) [noise=true , address=45.91.116.8:4047 , piece_id=YSK2OEGR7TG6CULCMG7YWASIFKPLR2M3BYHXJGECEWCUZIGVK3JQ, seg_pos={0 0}, upload=523267]: Long tail cancellation due to optimal threshold Dec 30 12:25:57.986470513 (596h56m57.857553558s) [seg_pos={0 0}, upload=523267]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:25:57.986551988 (596h56m57.857635033s) [upload=523267]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:25:57.986559051 (596h56m57.857642096s) [upload=523267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:25:58.279694175 (596h56m58.15077722s) [upload=523267]: Done uploading segments... Dec 30 12:25:58.279734552 (596h56m58.150817597s) [upload=523267]: Upload finished. err: <nil> Duration: 30.003514743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX.] duration: 3.682864392s trace: https://jaeger.dev.storj.io/trace/b1541968540caef FAIL (2025-12-30 12:26:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.682892184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.................X..................X.....X.] duration: 46.731285538s trace: https://jaeger.dev.storj.io/trace/66182f18a13b9083 FAIL (2025-12-30 12:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4:29.218557586 (596h55m29.089640621s) [noise=true , address=77.90.31.14:4683 , piece_id=6Y52NNSFVBFKMZZZ6O2LI5RNYB2RE6GI6DISHS7EHGVXMLPXAKVQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.218614705 (596h55m29.08969774s) [noise=true , address=77.90.31.19:4851 , piece_id=6EYEEIBKR4LWD3HFXYMYSCLFHMBD2RGV5RHIETAO3BI55ACBT3QA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219063640 (596h55m29.090146685s) [noise=true , address=145.79.227.4:4209 , piece_id=N46WW4XZHMHCFW3CZZZKI3MZ6WLI73ZW22SZ6NWB2JYQBMQHFG2Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219197745 (596h55m29.09028078s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219518305 (596h55m29.090601341s) [noise=true , address=77.90.31.15:4729 , piece_id=D5CC54BZLSRU6QDSQY74MVIV3WXK24WRRMAEPGMJYEM5M6KTMIYA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219969745 (596h55m29.09105278s) [noise=true , address=145.79.227.30:4771 , piece_id=NOUPDE42CLKP2REMD7B7UI7AE3X2Z2B6PZSKYAMEVFIZ7GYJT6ZA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.220383443 (596h55m29.091466488s) [noise=true , address=145.79.227.2:4118 , piece_id=JPRVX5R6OZHSNF7MXBNB2DFD4LYPO2RI3VOVEOKINSLNQDHG6WXA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.220705596 (596h55m29.091788641s) [noise=true , address=45.91.116.13:4179 , piece_id=QZQX3YZ6UCGYJ2WALMCMD5JV7JNR362YOSJXOS555OVF7R6H6UOQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.221040584 (596h55m29.092123629s) [noise=true , address=145.79.227.34:4938 , piece_id=ZITRNQEECQMGPEKIVO36P7CPMP6M6VV6ZKWOGTLO4P6SOXDK5L6A, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.221388637 (596h55m29.092471682s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222031882 (596h55m29.093114947s) [noise=true , address=145.79.227.40:4054 , piece_id=67OO7GDCYEAROSX4M45PXCZMNANX3BDUBKFVC6ZXR7KFW2ILCKPA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222427444 (596h55m29.093510479s) [noise=true , address=77.90.31.17:4782 , piece_id=VACJHSA7BDYV2BXZFQ2BLV7GEA3IGVJ4RTFB6KUWVGMZ6QTZ2EYQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222932376 (596h55m29.094015411s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.223141825 (596h55m29.09422486s) [noise=true , address=145.79.227.44:4220 , piece_id=5DO5QRYI57PTKTDXWUUNUFX3EGWHLAV7NRZ45XI6UH45UT34XOLA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.223588496 (596h55m29.094671541s) [noise=true , address=45.91.116.14:4205 , piece_id=P2HQGT24UDTSLV25WEAKVGKFOQCEBHJWP5TAL3RJXJXIFOUFBJ2A, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.224021911 (596h55m29.095104956s) [noise=true , address=45.91.116.6:4979 , piece_id=YGEKIRPFAHFFK3OETG3VS4RW4B47YEMR2NWYW4G5AS3MTAPDR4PA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.224373069 (596h55m29.095456114s) [noise=true , address=145.79.227.35:4510 , piece_id=6AFAZZ6L4HYQ2WOD6OHCM54JIDCIWUTHMKAFKXHAX6EF77WULC7Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:30.618334414 (596h55m30.489417459s) [upload=523245]: Next returned nil segment Dec 30 12:24:30.618353932 (596h55m30.489437047s) [upload=523245]: Waiting for error group managing segments... Dec 30 12:24:31.054823155 (596h55m30.92590619s) [noise=true , address=145.79.227.2:4118 , piece_id=JPRVX5R6OZHSNF7MXBNB2DFD4LYPO2RI3VOVEOKINSLNQDHG6WXA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.066436138 (596h55m30.937519173s) [noise=true , address=145.79.227.4:4209 , piece_id=N46WW4XZHMHCFW3CZZZKI3MZ6WLI73ZW22SZ6NWB2JYQBMQHFG2Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.257975509 (596h55m31.129058564s) [noise=true , address=77.90.31.17:4782 , piece_id=VACJHSA7BDYV2BXZFQ2BLV7GEA3IGVJ4RTFB6KUWVGMZ6QTZ2EYQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.359659290 (596h55m31.230742335s) [noise=true , address=77.90.31.14:4683 , piece_id=6Y52NNSFVBFKMZZZ6O2LI5RNYB2RE6GI6DISHS7EHGVXMLPXAKVQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.451112330 (596h55m31.322195375s) [noise=true , address=77.90.31.15:4729 , piece_id=D5CC54BZLSRU6QDSQY74MVIV3WXK24WRRMAEPGMJYEM5M6KTMIYA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.456017441 (596h55m31.327100476s) [noise=true , address=77.90.31.19:4851 , piece_id=6EYEEIBKR4LWD3HFXYMYSCLFHMBD2RGV5RHIETAO3BI55ACBT3QA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.463041835 (596h55m31.334124881s) [noise=true , address=45.91.116.14:4205 , piece_id=P2HQGT24UDTSLV25WEAKVGKFOQCEBHJWP5TAL3RJXJXIFOUFBJ2A, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.472219902 (596h55m31.343302948s) [noise=true , address=45.91.116.6:4979 , piece_id=YGEKIRPFAHFFK3OETG3VS4RW4B47YEMR2NWYW4G5AS3MTAPDR4PA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.512189029 (596h55m31.383272074s) [noise=true , address=45.91.116.13:4179 , piece_id=QZQX3YZ6UCGYJ2WALMCMD5JV7JNR362YOSJXOS555OVF7R6H6UOQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.999604142 (596h55m31.870687207s) [noise=true , address=145.79.227.44:4220 , piece_id=5DO5QRYI57PTKTDXWUUNUFX3EGWHLAV7NRZ45XI6UH45UT34XOLA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.004081829 (596h55m31.875164874s) [noise=true , address=145.79.227.30:4771 , piece_id=NOUPDE42CLKP2REMD7B7UI7AE3X2Z2B6PZSKYAMEVFIZ7GYJT6ZA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.081066598 (596h55m31.952149634s) [noise=true , address=145.79.227.40:4054 , piece_id=67OO7GDCYEAROSX4M45PXCZMNANX3BDUBKFVC6ZXR7KFW2ILCKPA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.084212910 (596h55m31.955295945s) [noise=true , address=145.79.227.35:4510 , piece_id=6AFAZZ6L4HYQ2WOD6OHCM54JIDCIWUTHMKAFKXHAX6EF77WULC7Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.091519412 (596h55m31.962602447s) [noise=true , address=145.79.227.34:4938 , piece_id=ZITRNQEECQMGPEKIVO36P7CPMP6M6VV6ZKWOGTLO4P6SOXDK5L6A, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.091542967 (596h55m31.962626032s) [seg_pos={0 0}, upload=523245]: Segment reached optimal threshold of 14 pieces. Dec 30 12:24:32.091785179 (596h55m31.962868224s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091794767 (596h55m31.962877822s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.091817650 (596h55m31.962900686s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vUrprFmwZQWzLjMRuS5b98w2owi4ZkAWKb6mU38DWhF61u7A4): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091825846 (596h55m31.962908881s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.091957076 (596h55m31.963040121s) [noise=true , address=45.91.116.8:4054 , piece_id=SWA4Z3WDCXUHU4BRQOHJOW7KJD35Y55BBNEN33IML2BCJYX2CDEA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091966644 (596h55m31.963049689s) [noise=true , address=45.91.116.8:4054 , piece_id=SWA4Z3WDCXUHU4BRQOHJOW7KJD35Y55BBNEN33IML2BCJYX2CDEA, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.092158510 (596h55m31.963241545s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.092168929 (596h55m31.963251984s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.092219435 (596h55m31.96330247s) [seg_pos={0 0}, upload=523245]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:24:32.092307223 (596h55m31.963390258s) [upload=523245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:24:32.092314196 (596h55m31.963397271s) [upload=523245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:24:32.374299058 (596h55m32.245382093s) [upload=523245]: Done uploading segments... Dec 30 12:24:32.374324336 (596h55m32.245407382s) [upload=523245]: Upload finished. err: <nil> Duration: 46.731301698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.................X..................X.....X] duration: 1m0.004080055s trace: https://jaeger.dev.storj.io/trace/66182f18a13b9083 FAIL (2025-12-30 12:25:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4:29.218557586 (596h55m29.089640621s) [noise=true , address=77.90.31.14:4683 , piece_id=6Y52NNSFVBFKMZZZ6O2LI5RNYB2RE6GI6DISHS7EHGVXMLPXAKVQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.218614705 (596h55m29.08969774s) [noise=true , address=77.90.31.19:4851 , piece_id=6EYEEIBKR4LWD3HFXYMYSCLFHMBD2RGV5RHIETAO3BI55ACBT3QA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219063640 (596h55m29.090146685s) [noise=true , address=145.79.227.4:4209 , piece_id=N46WW4XZHMHCFW3CZZZKI3MZ6WLI73ZW22SZ6NWB2JYQBMQHFG2Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219197745 (596h55m29.09028078s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219518305 (596h55m29.090601341s) [noise=true , address=77.90.31.15:4729 , piece_id=D5CC54BZLSRU6QDSQY74MVIV3WXK24WRRMAEPGMJYEM5M6KTMIYA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.219969745 (596h55m29.09105278s) [noise=true , address=145.79.227.30:4771 , piece_id=NOUPDE42CLKP2REMD7B7UI7AE3X2Z2B6PZSKYAMEVFIZ7GYJT6ZA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.220383443 (596h55m29.091466488s) [noise=true , address=145.79.227.2:4118 , piece_id=JPRVX5R6OZHSNF7MXBNB2DFD4LYPO2RI3VOVEOKINSLNQDHG6WXA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.220705596 (596h55m29.091788641s) [noise=true , address=45.91.116.13:4179 , piece_id=QZQX3YZ6UCGYJ2WALMCMD5JV7JNR362YOSJXOS555OVF7R6H6UOQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.221040584 (596h55m29.092123629s) [noise=true , address=145.79.227.34:4938 , piece_id=ZITRNQEECQMGPEKIVO36P7CPMP6M6VV6ZKWOGTLO4P6SOXDK5L6A, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.221388637 (596h55m29.092471682s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222031882 (596h55m29.093114947s) [noise=true , address=145.79.227.40:4054 , piece_id=67OO7GDCYEAROSX4M45PXCZMNANX3BDUBKFVC6ZXR7KFW2ILCKPA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222427444 (596h55m29.093510479s) [noise=true , address=77.90.31.17:4782 , piece_id=VACJHSA7BDYV2BXZFQ2BLV7GEA3IGVJ4RTFB6KUWVGMZ6QTZ2EYQ, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.222932376 (596h55m29.094015411s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.223141825 (596h55m29.09422486s) [noise=true , address=145.79.227.44:4220 , piece_id=5DO5QRYI57PTKTDXWUUNUFX3EGWHLAV7NRZ45XI6UH45UT34XOLA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.223588496 (596h55m29.094671541s) [noise=true , address=45.91.116.14:4205 , piece_id=P2HQGT24UDTSLV25WEAKVGKFOQCEBHJWP5TAL3RJXJXIFOUFBJ2A, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.224021911 (596h55m29.095104956s) [noise=true , address=45.91.116.6:4979 , piece_id=YGEKIRPFAHFFK3OETG3VS4RW4B47YEMR2NWYW4G5AS3MTAPDR4PA, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:29.224373069 (596h55m29.095456114s) [noise=true , address=145.79.227.35:4510 , piece_id=6AFAZZ6L4HYQ2WOD6OHCM54JIDCIWUTHMKAFKXHAX6EF77WULC7Q, seg_pos={0 0}, upload=523245]: Uploading piece... Dec 30 12:24:30.618334414 (596h55m30.489417459s) [upload=523245]: Next returned nil segment Dec 30 12:24:30.618353932 (596h55m30.489437047s) [upload=523245]: Waiting for error group managing segments... Dec 30 12:24:31.054823155 (596h55m30.92590619s) [noise=true , address=145.79.227.2:4118 , piece_id=JPRVX5R6OZHSNF7MXBNB2DFD4LYPO2RI3VOVEOKINSLNQDHG6WXA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.066436138 (596h55m30.937519173s) [noise=true , address=145.79.227.4:4209 , piece_id=N46WW4XZHMHCFW3CZZZKI3MZ6WLI73ZW22SZ6NWB2JYQBMQHFG2Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.257975509 (596h55m31.129058564s) [noise=true , address=77.90.31.17:4782 , piece_id=VACJHSA7BDYV2BXZFQ2BLV7GEA3IGVJ4RTFB6KUWVGMZ6QTZ2EYQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.359659290 (596h55m31.230742335s) [noise=true , address=77.90.31.14:4683 , piece_id=6Y52NNSFVBFKMZZZ6O2LI5RNYB2RE6GI6DISHS7EHGVXMLPXAKVQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.451112330 (596h55m31.322195375s) [noise=true , address=77.90.31.15:4729 , piece_id=D5CC54BZLSRU6QDSQY74MVIV3WXK24WRRMAEPGMJYEM5M6KTMIYA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.456017441 (596h55m31.327100476s) [noise=true , address=77.90.31.19:4851 , piece_id=6EYEEIBKR4LWD3HFXYMYSCLFHMBD2RGV5RHIETAO3BI55ACBT3QA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.463041835 (596h55m31.334124881s) [noise=true , address=45.91.116.14:4205 , piece_id=P2HQGT24UDTSLV25WEAKVGKFOQCEBHJWP5TAL3RJXJXIFOUFBJ2A, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.472219902 (596h55m31.343302948s) [noise=true , address=45.91.116.6:4979 , piece_id=YGEKIRPFAHFFK3OETG3VS4RW4B47YEMR2NWYW4G5AS3MTAPDR4PA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.512189029 (596h55m31.383272074s) [noise=true , address=45.91.116.13:4179 , piece_id=QZQX3YZ6UCGYJ2WALMCMD5JV7JNR362YOSJXOS555OVF7R6H6UOQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:31.999604142 (596h55m31.870687207s) [noise=true , address=145.79.227.44:4220 , piece_id=5DO5QRYI57PTKTDXWUUNUFX3EGWHLAV7NRZ45XI6UH45UT34XOLA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.004081829 (596h55m31.875164874s) [noise=true , address=145.79.227.30:4771 , piece_id=NOUPDE42CLKP2REMD7B7UI7AE3X2Z2B6PZSKYAMEVFIZ7GYJT6ZA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.081066598 (596h55m31.952149634s) [noise=true , address=145.79.227.40:4054 , piece_id=67OO7GDCYEAROSX4M45PXCZMNANX3BDUBKFVC6ZXR7KFW2ILCKPA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.084212910 (596h55m31.955295945s) [noise=true , address=145.79.227.35:4510 , piece_id=6AFAZZ6L4HYQ2WOD6OHCM54JIDCIWUTHMKAFKXHAX6EF77WULC7Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.091519412 (596h55m31.962602447s) [noise=true , address=145.79.227.34:4938 , piece_id=ZITRNQEECQMGPEKIVO36P7CPMP6M6VV6ZKWOGTLO4P6SOXDK5L6A, seg_pos={0 0}, upload=523245]: Done uploading piece. err: <nil> Dec 30 12:24:32.091542967 (596h55m31.962626032s) [seg_pos={0 0}, upload=523245]: Segment reached optimal threshold of 14 pieces. Dec 30 12:24:32.091785179 (596h55m31.962868224s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091794767 (596h55m31.962877822s) [noise=true , address=193.149.164.143:4698 , piece_id=Y7LCB37SKX4AK4L7OSUBVXTXCJBI5G5RYCOFMPUJGWLAMAT6GC5Q, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.091817650 (596h55m31.962900686s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vUrprFmwZQWzLjMRuS5b98w2owi4ZkAWKb6mU38DWhF61u7A4): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091825846 (596h55m31.962908881s) [noise=true , address=45.91.116.11:4136 , piece_id=GF4KYBJPRESKL25UUHLZO37QUXG6FUDHIRMQ3CN2G7XYH5ZOGTEA, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.091957076 (596h55m31.963040121s) [noise=true , address=45.91.116.8:4054 , piece_id=SWA4Z3WDCXUHU4BRQOHJOW7KJD35Y55BBNEN33IML2BCJYX2CDEA, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.091966644 (596h55m31.963049689s) [noise=true , address=45.91.116.8:4054 , piece_id=SWA4Z3WDCXUHU4BRQOHJOW7KJD35Y55BBNEN33IML2BCJYX2CDEA, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.092158510 (596h55m31.963241545s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY): protocol: expected piece hash; context canceled; EOF Dec 30 12:24:32.092168929 (596h55m31.963251984s) [noise=true , address=145.79.227.43:4192 , piece_id=3VWZO2T6ENOYJ2OWQG5S2NQAOSZVPLFW35IYNRRX7BJW6SI4IUDQ, seg_pos={0 0}, upload=523245]: Long tail cancellation due to optimal threshold Dec 30 12:24:32.092219435 (596h55m31.96330247s) [seg_pos={0 0}, upload=523245]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:24:32.092307223 (596h55m31.963390258s) [upload=523245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:24:32.092314196 (596h55m31.963397271s) [upload=523245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:24:32.374299058 (596h55m32.245382093s) [upload=523245]: Done uploading segments... Dec 30 12:24:32.374324336 (596h55m32.245407382s) [upload=523245]: Upload finished. err: <nil> Duration: 1m0.004091717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X...X..X..X.X....X.X.......XX......XX.XX....XX..X.......] duration: 6.097242208s trace: https://jaeger.dev.storj.io/trace/55650e95b5e50541 FAIL (2025-12-30 12:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA) context deadline exceeded ======LOGS====== Dec 30 12:19:45.766284145 (596h50m45.63736718s) [upload=523180]: Starting upload Dec 30 12:19:45.766340162 (596h50m45.637423197s) [upload=523180]: Done starting upload Dec 30 12:19:45.766454009 (596h50m45.637537054s) [upload=523180]: Uploading segments... Dec 30 12:19:45.766557346 (596h50m45.637640381s) [upload=523180]: Got next segment. Inline: false Dec 30 12:19:45.766624073 (596h50m45.637707108s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:19:45.766629303 (596h50m45.637712348s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:19:46.014347235 (596h50m45.88543027s) [seg_pos={0 0}, upload=523180]: Begin upload segment... Dec 30 12:19:46.014420124 (596h50m45.885503159s) [seg_pos={0 0}, upload=523180]: Done begin upload segment. Dec 30 12:19:46.014473766 (596h50m45.885556812s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014549330 (596h50m45.885632375s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014976032 (596h50m45.886059068s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015267518 (596h50m45.886350553s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015657480 (596h50m45.886740516s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015950638 (596h50m45.887033683s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016225201 (596h50m45.887308236s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016830384 (596h50m45.887913429s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.017285651 (596h50m45.888368686s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018246059 (596h50m45.889329094s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018668644 (596h50m45.889751679s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.020370645 (596h50m45.89145369s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.031374528 (596h50m45.902457563s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.034124705 (596h50m45.90520774s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.037330750 (596h50m45.908413825s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.038396720 (596h50m45.909479765s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:47.717911569 (596h50m47.588994604s) [upload=523180]: Next returned nil segment Dec 30 12:19:47.717971914 (596h50m47.589054959s) [upload=523180]: Waiting for error group managing segments... Dec 30 12:19:48.007728896 (596h50m47.878811941s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.010593751 (596h50m47.881676786s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.022380223 (596h50m47.893463269s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.028820647 (596h50m47.899903682s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.031967008 (596h50m47.903050043s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.032338145 (596h50m47.90342118s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.038366814 (596h50m47.909449859s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.042741945 (596h50m47.91382498s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.074672897 (596h50m47.945755932s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.089235796 (596h50m47.960318842s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.279503706 (596h50m48.150586741s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.432335881 (596h50m48.303418916s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484710199 (596h50m48.355793234s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484726912 (596h50m48.355809947s) [seg_pos={0 0}, upload=523180]: Segment reached optimal threshold of 13 pieces. Dec 30 12:19:48.484916462 (596h50m48.355999497s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.484923066 (596h50m48.356006101s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485100303 (596h50m48.356183338s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485106204 (596h50m48.356189239s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485163953 (596h50m48.356246988s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485180966 (596h50m48.356264001s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485207737 (596h50m48.356290772s) [seg_pos={0 0}, upload=523180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:19:48.485303609 (596h50m48.356386644s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:19:48.485310784 (596h50m48.356393819s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:19:48.781649176 (596h50m48.652732211s) [upload=523180]: Done uploading segments... Dec 30 12:19:48.781696327 (596h50m48.652779372s) [upload=523180]: Upload finished. err: <nil> Duration: 6.097266443s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XXX] duration: 5.001131488s trace: https://jaeger.dev.storj.io/trace/b1541968540caef FAIL (2025-12-30 12:25:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001155694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X...X..X..X.X....X.X.......XX......XX.XX....XX..X......] duration: 5.6575663s trace: https://jaeger.dev.storj.io/trace/55650e95b5e50541 FAIL (2025-12-30 12:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA) context deadline exceeded ======LOGS====== Dec 30 12:19:45.766284145 (596h50m45.63736718s) [upload=523180]: Starting upload Dec 30 12:19:45.766340162 (596h50m45.637423197s) [upload=523180]: Done starting upload Dec 30 12:19:45.766454009 (596h50m45.637537054s) [upload=523180]: Uploading segments... Dec 30 12:19:45.766557346 (596h50m45.637640381s) [upload=523180]: Got next segment. Inline: false Dec 30 12:19:45.766624073 (596h50m45.637707108s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:19:45.766629303 (596h50m45.637712348s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:19:46.014347235 (596h50m45.88543027s) [seg_pos={0 0}, upload=523180]: Begin upload segment... Dec 30 12:19:46.014420124 (596h50m45.885503159s) [seg_pos={0 0}, upload=523180]: Done begin upload segment. Dec 30 12:19:46.014473766 (596h50m45.885556812s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014549330 (596h50m45.885632375s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014976032 (596h50m45.886059068s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015267518 (596h50m45.886350553s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015657480 (596h50m45.886740516s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015950638 (596h50m45.887033683s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016225201 (596h50m45.887308236s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016830384 (596h50m45.887913429s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.017285651 (596h50m45.888368686s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018246059 (596h50m45.889329094s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018668644 (596h50m45.889751679s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.020370645 (596h50m45.89145369s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.031374528 (596h50m45.902457563s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.034124705 (596h50m45.90520774s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.037330750 (596h50m45.908413825s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.038396720 (596h50m45.909479765s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:47.717911569 (596h50m47.588994604s) [upload=523180]: Next returned nil segment Dec 30 12:19:47.717971914 (596h50m47.589054959s) [upload=523180]: Waiting for error group managing segments... Dec 30 12:19:48.007728896 (596h50m47.878811941s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.010593751 (596h50m47.881676786s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.022380223 (596h50m47.893463269s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.028820647 (596h50m47.899903682s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.031967008 (596h50m47.903050043s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.032338145 (596h50m47.90342118s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.038366814 (596h50m47.909449859s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.042741945 (596h50m47.91382498s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.074672897 (596h50m47.945755932s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.089235796 (596h50m47.960318842s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.279503706 (596h50m48.150586741s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.432335881 (596h50m48.303418916s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484710199 (596h50m48.355793234s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484726912 (596h50m48.355809947s) [seg_pos={0 0}, upload=523180]: Segment reached optimal threshold of 13 pieces. Dec 30 12:19:48.484916462 (596h50m48.355999497s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.484923066 (596h50m48.356006101s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485100303 (596h50m48.356183338s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485106204 (596h50m48.356189239s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485163953 (596h50m48.356246988s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485180966 (596h50m48.356264001s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485207737 (596h50m48.356290772s) [seg_pos={0 0}, upload=523180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:19:48.485303609 (596h50m48.356386644s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:19:48.485310784 (596h50m48.356393819s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:19:48.781649176 (596h50m48.652732211s) [upload=523180]: Done uploading segments... Dec 30 12:19:48.781696327 (596h50m48.652779372s) [upload=523180]: Upload finished. err: <nil> Duration: 5.657596949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.XX] duration: 5.00068429s trace: https://jaeger.dev.storj.io/trace/dca3551ec578be6 FAIL (2025-12-30 12:24:39 UTC) orbiter warning: context deadline exceeded Duration: 5.000707464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X...X..X..X.X....X.X.......XX......XX.XX....XX..X.....] duration: 5.878209662s trace: https://jaeger.dev.storj.io/trace/55650e95b5e50541 FAIL (2025-12-30 12:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA) context deadline exceeded ======LOGS====== Dec 30 12:19:45.766284145 (596h50m45.63736718s) [upload=523180]: Starting upload Dec 30 12:19:45.766340162 (596h50m45.637423197s) [upload=523180]: Done starting upload Dec 30 12:19:45.766454009 (596h50m45.637537054s) [upload=523180]: Uploading segments... Dec 30 12:19:45.766557346 (596h50m45.637640381s) [upload=523180]: Got next segment. Inline: false Dec 30 12:19:45.766624073 (596h50m45.637707108s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:19:45.766629303 (596h50m45.637712348s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:19:46.014347235 (596h50m45.88543027s) [seg_pos={0 0}, upload=523180]: Begin upload segment... Dec 30 12:19:46.014420124 (596h50m45.885503159s) [seg_pos={0 0}, upload=523180]: Done begin upload segment. Dec 30 12:19:46.014473766 (596h50m45.885556812s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014549330 (596h50m45.885632375s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.014976032 (596h50m45.886059068s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015267518 (596h50m45.886350553s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015657480 (596h50m45.886740516s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.015950638 (596h50m45.887033683s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016225201 (596h50m45.887308236s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.016830384 (596h50m45.887913429s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.017285651 (596h50m45.888368686s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018246059 (596h50m45.889329094s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.018668644 (596h50m45.889751679s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.020370645 (596h50m45.89145369s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.031374528 (596h50m45.902457563s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.034124705 (596h50m45.90520774s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.037330750 (596h50m45.908413825s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:46.038396720 (596h50m45.909479765s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Uploading piece... Dec 30 12:19:47.717911569 (596h50m47.588994604s) [upload=523180]: Next returned nil segment Dec 30 12:19:47.717971914 (596h50m47.589054959s) [upload=523180]: Waiting for error group managing segments... Dec 30 12:19:48.007728896 (596h50m47.878811941s) [noise=true , address=145.79.227.41:4086 , piece_id=RNDZOTWE6IFCENVGCI4MSWHW52IHUS7YALS5XN4DF6I3MOJLWLZQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.010593751 (596h50m47.881676786s) [noise=true , address=145.79.227.3:4149 , piece_id=SAGCE6BE2GM2K7CVTYIWOB4OEV4VCLHE2QCOVPWMSUYR3NSDUNPQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.022380223 (596h50m47.893463269s) [noise=true , address=145.79.227.46:4601 , piece_id=SW7ZX3PVI7BQ6MCCIQ2CYYJZHY4DKSFQKFJZC2KVYL3TRKJ3QYHA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.028820647 (596h50m47.899903682s) [noise=true , address=145.79.227.16:4910 , piece_id=ABUT7A3RMCL5Q5SUYI3IJ5CS3CXYV43AWPXQGCXKY3VQQHLSF4SQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.031967008 (596h50m47.903050043s) [noise=true , address=145.79.227.42:4122 , piece_id=VYADST6SPHVENQ7RBPWNDM4RV4UZY7PBHZZ7UK33JEAF6RGJWD3Q, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.032338145 (596h50m47.90342118s) [noise=true , address=145.79.227.17:4971 , piece_id=P7CL2MRCU6LCF3MN3T2M6YRPI7O3CD4MODN5P4UBSON3V22IJRYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.038366814 (596h50m47.909449859s) [noise=true , address=145.79.227.45:4229 , piece_id=VXEWILI6PAJJSCNGNRWR5IUKUCBQCBJ4LXVAGUF2M3ILSMPMLVGQ, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.042741945 (596h50m47.91382498s) [noise=true , address=145.79.227.36:4519 , piece_id=YN2CWSUMI4WNH47IIOQ6TEPFZV45BPS7Q2NA5UMKT2PPW4R7DZYA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.074672897 (596h50m47.945755932s) [noise=true , address=145.79.227.11:4731 , piece_id=DXJ2LNZUWHSDJS6DGNIJM3N6QFLL532PH4Z3CVEXBK2YSXMHGLBA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.089235796 (596h50m47.960318842s) [noise=true , address=145.79.227.2:4114 , piece_id=GSEIOMNEXFML2NB6E2MAPJQY62Q5CV3VTFVLSELNICRONSY6H6SA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.279503706 (596h50m48.150586741s) [noise=true , address=77.90.31.21:4878 , piece_id=QM46SJ3WJRX5U66G4MYGRG5QFLAXO6GJIEGI3GSS66STIDC5KRPA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.432335881 (596h50m48.303418916s) [noise=true , address=45.91.116.13:4156 , piece_id=RZA6TCL5J6VKTL6NCVVBTD6L2IPBJDTNYFGRLB6EELEULLQJW2AA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484710199 (596h50m48.355793234s) [noise=true , address=77.90.31.10:4519 , piece_id=FC7TU5ICGIQRI6ASPSGMM5KVYHW6AKBLO6NPVL6GQXA3MM5SAX7A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: <nil> Dec 30 12:19:48.484726912 (596h50m48.355809947s) [seg_pos={0 0}, upload=523180]: Segment reached optimal threshold of 13 pieces. Dec 30 12:19:48.484916462 (596h50m48.355999497s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.484923066 (596h50m48.356006101s) [noise=true , address=45.91.116.8:4047 , piece_id=KSFHB4BYOE4AQNSUKBAKUPXYQB3E3SHPKVAXJKH6UDY6FDFOTL3A, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485100303 (596h50m48.356183338s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485106204 (596h50m48.356189239s) [noise=true , address=145.79.227.43:4158 , piece_id=FZPSP5JNKALU76V4SYKZGO4UBOXQSHK5BPVSNKTVNFWNYI4NECUA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485163953 (596h50m48.356246988s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:48.485180966 (596h50m48.356264001s) [noise=true , address=45.91.116.7:4010 , piece_id=EN6QMDUXGGWCXVFZ3NFHFK4NI4EPC5I26JNDAXWSYDIV7ZRB7TWA, seg_pos={0 0}, upload=523180]: Long tail cancellation due to optimal threshold Dec 30 12:19:48.485207737 (596h50m48.356290772s) [seg_pos={0 0}, upload=523180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:19:48.485303609 (596h50m48.356386644s) [upload=523180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:19:48.485310784 (596h50m48.356393819s) [upload=523180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:19:48.781649176 (596h50m48.652732211s) [upload=523180]: Done uploading segments... Dec 30 12:19:48.781696327 (596h50m48.652779372s) [upload=523180]: Upload finished. err: <nil> Duration: 5.878235421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X................X.................X..................X.....] duration: 40.74452825s trace: https://jaeger.dev.storj.io/trace/3553f4ed3f09eeb0 FAIL (2025-12-30 12:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 69087s) [noise=true , address=45.91.116.13:4179 , piece_id=CWP6OYVX5TVN3K6QNSDELAHYHDILDDX4OEPVY6OCLCGRLETYKVCA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.748569563 (596h50m32.619652608s) [noise=true , address=77.90.31.14:4678 , piece_id=VTPW6ERBG67ENHWCHZS2JJV7KXAK6FORFZUGFNK7AB4IQ4PKTZ2Q, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.749002146 (596h50m32.620085191s) [noise=true , address=64.49.13.169:4716 , piece_id=4UPP7GF2ODHGT2KZRNKA3WPROSCGTAJ4H75EQOMCMWYR5UOPCHYA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.749432927 (596h50m32.620515972s) [noise=true , address=145.79.227.49:4370 , piece_id=AGS7HXSLHQ35VEAJ6MJ5GRBR5K5RHCZBJJR4MJ6IJOOCDQSU2AQQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.749964409 (596h50m32.621047454s) [noise=true , address=145.79.227.44:4198 , piece_id=JIQVYPEXDA4IONPP7J3G3UM4VUHNUB344CFTXPAZQWK6HMZNHPSQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.750373959 (596h50m32.621457004s) [noise=true , address=193.149.164.143:4694 , piece_id=IZ2XZQVXF2C3KSYBY4YNX5T2YG53XM3V44OEQ34M477SSVJN7VLA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.750796213 (596h50m32.621879248s) [noise=true , address=64.49.13.26:4949 , piece_id=UIANYXE3U47NJALST3V64ETXLMSSUMRKIYR6SYR36XCXGHA7U7VQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.751262610 (596h50m32.622345655s) [noise=true , address=145.79.227.40:4076 , piece_id=SLGKBIOZ5YZPIMHLLNTLN34D6Q3VXYBJVJANFZNUUXG4VQ3JSARA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.752667957 (596h50m32.623751002s) [noise=true , address=77.90.31.10:4533 , piece_id=6RXKYKAHG3ZRN7CA3ED6JG6OEDXJEGW5JWN5OC5QN5LP6SIZANDA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.754144488 (596h50m32.625227533s) [noise=true , address=77.90.31.18:4816 , piece_id=GFLBHT54EL7OI2TJSAXXPCNVY5WANORUQQ4FAT5QGH3P4XX4EO4Q, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.756148845 (596h50m32.62723188s) [noise=true , address=77.90.31.17:4790 , piece_id=V3CHFTJOLNTCUK25YNEVS36FHSEFYRBHOSNKHDIDQ4KEHLXNN7JQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.758270545 (596h50m32.62935358s) [noise=true , address=45.91.116.14:4220 , piece_id=KZYLP2TG5FHO5LKTFYRBGVSP7PPD4DHHSFD6WA2WNKW2NHTGFBMQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.760224626 (596h50m32.631307671s) [noise=true , address=145.79.227.33:4879 , piece_id=K2WMATKLYHB3A7O5ZSOM3FBQEH6BIH63PPXCFGN34VZYNSKIA4LQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.761791510 (596h50m32.632874545s) [noise=true , address=77.90.31.21:4909 , piece_id=N4DRY6DURLMEBJ6E2PM3257LQ4Q2Y2KODES6AT6NCQZJKCB6DLOQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.762860515 (596h50m32.63394356s) [noise=true , address=145.79.227.30:4780 , piece_id=AK45WLLG7BHNT2J6LUFMVBXLSM43UAJUDMB2UOX35COA475H7BZQ, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:32.763412847 (596h50m32.634495882s) [noise=true , address=77.90.31.15:4716 , piece_id=R7C5CLSIAUME2OUJDINMJPMPCOQRBTYBDF4MWDKO7FCXBYRHIQQA, seg_pos={0 0}, upload=523174]: Uploading piece... Dec 30 12:19:34.192315925 (596h50m34.06339896s) [upload=523174]: Next returned nil segment Dec 30 12:19:34.192327867 (596h50m34.063410912s) [upload=523174]: Waiting for error group managing segments... Dec 30 12:19:34.560819872 (596h50m34.431902917s) [noise=true , address=64.49.13.26:4949 , piece_id=UIANYXE3U47NJALST3V64ETXLMSSUMRKIYR6SYR36XCXGHA7U7VQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:34.567668473 (596h50m34.438751508s) [noise=true , address=64.49.13.155:4916 , piece_id=J6BEBCVSYOHC6CZZGJSUBFI4E2AM46ZJ222UON2QCRF3YTS2KWTQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:34.802060120 (596h50m34.673143155s) [noise=true , address=77.90.31.14:4678 , piece_id=VTPW6ERBG67ENHWCHZS2JJV7KXAK6FORFZUGFNK7AB4IQ4PKTZ2Q, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:34.812985725 (596h50m34.68406877s) [noise=true , address=77.90.31.21:4909 , piece_id=N4DRY6DURLMEBJ6E2PM3257LQ4Q2Y2KODES6AT6NCQZJKCB6DLOQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:34.913461155 (596h50m34.78454422s) [noise=true , address=45.91.116.14:4220 , piece_id=KZYLP2TG5FHO5LKTFYRBGVSP7PPD4DHHSFD6WA2WNKW2NHTGFBMQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:34.929197459 (596h50m34.800280494s) [noise=true , address=77.90.31.17:4790 , piece_id=V3CHFTJOLNTCUK25YNEVS36FHSEFYRBHOSNKHDIDQ4KEHLXNN7JQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.004040141 (596h50m34.875123186s) [noise=true , address=77.90.31.18:4816 , piece_id=GFLBHT54EL7OI2TJSAXXPCNVY5WANORUQQ4FAT5QGH3P4XX4EO4Q, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.103494477 (596h50m34.974577512s) [noise=true , address=45.91.116.13:4179 , piece_id=CWP6OYVX5TVN3K6QNSDELAHYHDILDDX4OEPVY6OCLCGRLETYKVCA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.488741085 (596h50m35.35982413s) [noise=true , address=145.79.227.44:4198 , piece_id=JIQVYPEXDA4IONPP7J3G3UM4VUHNUB344CFTXPAZQWK6HMZNHPSQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.499900023 (596h50m35.370983068s) [noise=true , address=145.79.227.30:4780 , piece_id=AK45WLLG7BHNT2J6LUFMVBXLSM43UAJUDMB2UOX35COA475H7BZQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.557728768 (596h50m35.428811803s) [noise=true , address=145.79.227.33:4879 , piece_id=K2WMATKLYHB3A7O5ZSOM3FBQEH6BIH63PPXCFGN34VZYNSKIA4LQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.571312994 (596h50m35.442396049s) [noise=true , address=145.79.227.34:4929 , piece_id=RZVMOBGUSNK7HVVVOOBT4VK5DLVLQP22ZZUW4I75G36VRG3YCS7A, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.596929562 (596h50m35.468012597s) [noise=true , address=145.79.227.40:4076 , piece_id=SLGKBIOZ5YZPIMHLLNTLN34D6Q3VXYBJVJANFZNUUXG4VQ3JSARA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.639958555 (596h50m35.5110416s) [noise=true , address=77.90.31.15:4716 , piece_id=R7C5CLSIAUME2OUJDINMJPMPCOQRBTYBDF4MWDKO7FCXBYRHIQQA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: <nil> Dec 30 12:19:35.639980978 (596h50m35.511064024s) [seg_pos={0 0}, upload=523174]: Segment reached optimal threshold of 14 pieces. Dec 30 12:19:35.640234621 (596h50m35.511317686s) [noise=true , address=193.149.164.143:4694 , piece_id=IZ2XZQVXF2C3KSYBY4YNX5T2YG53XM3V44OEQ34M477SSVJN7VLA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:35.640246323 (596h50m35.511329358s) [noise=true , address=193.149.164.143:4694 , piece_id=IZ2XZQVXF2C3KSYBY4YNX5T2YG53XM3V44OEQ34M477SSVJN7VLA, seg_pos={0 0}, upload=523174]: Long tail cancellation due to optimal threshold Dec 30 12:19:35.640257505 (596h50m35.51134054s) [noise=true , address=145.79.227.49:4370 , piece_id=AGS7HXSLHQ35VEAJ6MJ5GRBR5K5RHCZBJJR4MJ6IJOOCDQSU2AQQ, seg_pos={0 0}, upload=523174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:35.640262985 (596h50m35.51134602s) [noise=true , address=145.79.227.49:4370 , piece_id=AGS7HXSLHQ35VEAJ6MJ5GRBR5K5RHCZBJJR4MJ6IJOOCDQSU2AQQ, seg_pos={0 0}, upload=523174]: Long tail cancellation due to optimal threshold Dec 30 12:19:35.640381591 (596h50m35.511464646s) [noise=true , address=77.90.31.10:4533 , piece_id=6RXKYKAHG3ZRN7CA3ED6JG6OEDXJEGW5JWN5OC5QN5LP6SIZANDA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: expected piece hash; context canceled; EOF Dec 30 12:19:35.640390919 (596h50m35.511473954s) [noise=true , address=77.90.31.10:4533 , piece_id=6RXKYKAHG3ZRN7CA3ED6JG6OEDXJEGW5JWN5OC5QN5LP6SIZANDA, seg_pos={0 0}, upload=523174]: Long tail cancellation due to optimal threshold Dec 30 12:19:35.640581442 (596h50m35.511664487s) [noise=true , address=64.49.13.169:4716 , piece_id=4UPP7GF2ODHGT2KZRNKA3WPROSCGTAJ4H75EQOMCMWYR5UOPCHYA, seg_pos={0 0}, upload=523174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:34558->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:34558->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34558->64.49.13.169:4716: use of closed network connection Dec 30 12:19:35.640589757 (596h50m35.511672802s) [noise=true , address=64.49.13.169:4716 , piece_id=4UPP7GF2ODHGT2KZRNKA3WPROSCGTAJ4H75EQOMCMWYR5UOPCHYA, seg_pos={0 0}, upload=523174]: Long tail cancellation due to optimal threshold Dec 30 12:19:35.640620255 (596h50m35.51170329s) [seg_pos={0 0}, upload=523174]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:19:35.640703353 (596h50m35.511786388s) [upload=523174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:19:35.640709917 (596h50m35.511792992s) [upload=523174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:19:35.933458284 (596h50m35.804541329s) [upload=523174]: Done uploading segments... Dec 30 12:19:35.933477059 (596h50m35.804560095s) [upload=523174]: Upload finished. err: <nil> Duration: 40.744552787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XXX.XX.X] duration: 5.001745789s trace: https://jaeger.dev.storj.io/trace/188be7cc309985c3 FAIL (2025-12-30 12:23:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001767982s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)