Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X........X..............................................] duration: 5.000969574s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:52:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000994692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X....X........X.............................................] duration: 4.275883644s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:51:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.27590275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..] duration: 5.971674963s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:51:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 5.971700924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X.] duration: 6.046296485s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:51:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 6.046323547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X....X........X............................................] duration: 3.488787055s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:50:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488816131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X] duration: 30.003444139s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 30.003466953s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.X....X........X...........................................] duration: 3.554533358s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:50:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555911873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..] duration: 6.058948307s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 6.058974018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.X....X........X..........................................] duration: 3.533406098s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:49:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533421106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X.] duration: 5.803523916s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:49:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 5.803553152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.X....X........X.........................................] duration: 4.337885736s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:48:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.337914942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X] duration: 30.00377436s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 30.003799146s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.X....X........X........................................] duration: 3.426454372s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:48:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426483058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.] duration: 6.031374825s trace: https://jaeger.dev.storj.io/trace/50de85881f962460 FAIL (2025-12-30 05:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA) context deadline exceeded ======LOGS====== 0 05:46:53.012795453 (590h17m52.883878498s) [upload=517478]: Starting upload Dec 30 05:46:53.012882198 (590h17m52.883965243s) [upload=517478]: Done starting upload Dec 30 05:46:53.012986818 (590h17m52.884069863s) [upload=517478]: Uploading segments... Dec 30 05:46:53.013325974 (590h17m52.884409019s) [upload=517478]: Got next segment. Inline: false Dec 30 05:46:53.013424400 (590h17m52.884507445s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:46:53.013432156 (590h17m52.884515201s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:46:53.257491908 (590h17m53.128574943s) [seg_pos={0 0}, upload=517478]: Begin upload segment... Dec 30 05:46:53.257600905 (590h17m53.12868395s) [seg_pos={0 0}, upload=517478]: Done begin upload segment. Dec 30 05:46:53.257671419 (590h17m53.128754464s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258797032 (590h17m53.129880077s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258854192 (590h17m53.129937237s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259449876 (590h17m53.130532921s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259491024 (590h17m53.130574069s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259994784 (590h17m53.131077829s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260292610 (590h17m53.131375655s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260621357 (590h17m53.131704402s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260794997 (590h17m53.131878042s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261162797 (590h17m53.132245832s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261378388 (590h17m53.132461433s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261697545 (590h17m53.132780611s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262190164 (590h17m53.133273209s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262402448 (590h17m53.133485483s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263016006 (590h17m53.134099051s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263558349 (590h17m53.134641384s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:54.965961831 (590h17m54.837044866s) [upload=517478]: Next returned nil segment Dec 30 05:46:54.965975406 (590h17m54.837058441s) [upload=517478]: Waiting for error group managing segments... Dec 30 05:46:55.245851740 (590h17m55.116934775s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.246456000 (590h17m55.117539045s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.254156313 (590h17m55.125239348s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.269694441 (590h17m55.140777476s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.270822809 (590h17m55.141905844s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.273548709 (590h17m55.144631744s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.274473851 (590h17m55.145556896s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.329920050 (590h17m55.201003086s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.330106185 (590h17m55.20118924s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.331420558 (590h17m55.202503593s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.343200498 (590h17m55.214283543s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.431556082 (590h17m55.302639127s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562019638 (590h17m55.433102683s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562039837 (590h17m55.433122872s) [seg_pos={0 0}, upload=517478]: Segment reached optimal threshold of 13 pieces. Dec 30 05:46:55.562216062 (590h17m55.433299097s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562222053 (590h17m55.433305098s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562459466 (590h17m55.433542511s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection Dec 30 05:46:55.562467961 (590h17m55.433551007s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562478351 (590h17m55.433561396s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562485125 (590h17m55.43356816s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562502237 (590h17m55.433585282s) [seg_pos={0 0}, upload=517478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:46:55.562568784 (590h17m55.433651819s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:46:55.562573392 (590h17m55.433656427s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:46:55.846330195 (590h17m55.71741323s) [upload=517478]: Done uploading segments... Dec 30 05:46:55.846408714 (590h17m55.717491749s) [upload=517478]: Upload finished. err: <nil> Duration: 6.031494963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX] duration: 30.002535059s trace: https://jaeger.dev.storj.io/trace/50de85881f962460 FAIL (2025-12-30 05:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA) context deadline exceeded ======LOGS====== 0 05:46:53.012795453 (590h17m52.883878498s) [upload=517478]: Starting upload Dec 30 05:46:53.012882198 (590h17m52.883965243s) [upload=517478]: Done starting upload Dec 30 05:46:53.012986818 (590h17m52.884069863s) [upload=517478]: Uploading segments... Dec 30 05:46:53.013325974 (590h17m52.884409019s) [upload=517478]: Got next segment. Inline: false Dec 30 05:46:53.013424400 (590h17m52.884507445s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:46:53.013432156 (590h17m52.884515201s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:46:53.257491908 (590h17m53.128574943s) [seg_pos={0 0}, upload=517478]: Begin upload segment... Dec 30 05:46:53.257600905 (590h17m53.12868395s) [seg_pos={0 0}, upload=517478]: Done begin upload segment. Dec 30 05:46:53.257671419 (590h17m53.128754464s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258797032 (590h17m53.129880077s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258854192 (590h17m53.129937237s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259449876 (590h17m53.130532921s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259491024 (590h17m53.130574069s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259994784 (590h17m53.131077829s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260292610 (590h17m53.131375655s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260621357 (590h17m53.131704402s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260794997 (590h17m53.131878042s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261162797 (590h17m53.132245832s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261378388 (590h17m53.132461433s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261697545 (590h17m53.132780611s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262190164 (590h17m53.133273209s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262402448 (590h17m53.133485483s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263016006 (590h17m53.134099051s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263558349 (590h17m53.134641384s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:54.965961831 (590h17m54.837044866s) [upload=517478]: Next returned nil segment Dec 30 05:46:54.965975406 (590h17m54.837058441s) [upload=517478]: Waiting for error group managing segments... Dec 30 05:46:55.245851740 (590h17m55.116934775s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.246456000 (590h17m55.117539045s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.254156313 (590h17m55.125239348s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.269694441 (590h17m55.140777476s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.270822809 (590h17m55.141905844s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.273548709 (590h17m55.144631744s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.274473851 (590h17m55.145556896s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.329920050 (590h17m55.201003086s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.330106185 (590h17m55.20118924s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.331420558 (590h17m55.202503593s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.343200498 (590h17m55.214283543s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.431556082 (590h17m55.302639127s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562019638 (590h17m55.433102683s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562039837 (590h17m55.433122872s) [seg_pos={0 0}, upload=517478]: Segment reached optimal threshold of 13 pieces. Dec 30 05:46:55.562216062 (590h17m55.433299097s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562222053 (590h17m55.433305098s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562459466 (590h17m55.433542511s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection Dec 30 05:46:55.562467961 (590h17m55.433551007s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562478351 (590h17m55.433561396s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562485125 (590h17m55.43356816s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562502237 (590h17m55.433585282s) [seg_pos={0 0}, upload=517478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:46:55.562568784 (590h17m55.433651819s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:46:55.562573392 (590h17m55.433656427s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:46:55.846330195 (590h17m55.71741323s) [upload=517478]: Done uploading segments... Dec 30 05:46:55.846408714 (590h17m55.717491749s) [upload=517478]: Upload finished. err: <nil> Duration: 30.002552453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.X....X........X.......................................] duration: 3.478179568s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:46:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478203072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.X] duration: 30.001928324s trace: https://jaeger.dev.storj.io/trace/2bc10f7f06d3a090 FAIL (2025-12-30 05:46:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: XRIN7Q24PQRFNT3JWRYZJDCZ2BUM2T26JIGWKGEL64EKLT73YKJA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: OR633MCN32BCTWPDSEYZJ4BW24VIOCTJQ6CGSGYEAANHSWSUN2OA) context deadline exceeded ======LOGS====== Dec 30 05:45:57.679979105 (590h16m57.55106214s) [upload=517464]: Starting upload Dec 30 05:45:57.680055079 (590h16m57.551138125s) [upload=517464]: Done starting upload Dec 30 05:45:57.680300898 (590h16m57.551383943s) [upload=517464]: Uploading segments... Dec 30 05:45:57.680363347 (590h16m57.551446382s) [upload=517464]: Got next segment. Inline: false Dec 30 05:45:57.680421978 (590h16m57.551505013s) [upload=517464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:45:57.680427529 (590h16m57.551510574s) [upload=517464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:45:57.927151944 (590h16m57.798234999s) [seg_pos={0 0}, upload=517464]: Begin upload segment... Dec 30 05:45:57.927247847 (590h16m57.798330892s) [seg_pos={0 0}, upload=517464]: Done begin upload segment. Dec 30 05:45:57.927308743 (590h16m57.798391788s) [noise=true , address=145.79.227.48:4302 , piece_id=3TULKKSZ2XZ5X6XLQ5CONK3XPORAZRPGRNJHN67IU7A6TL3JJ3KA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.927315366 (590h16m57.798398411s) [noise=true , address=77.90.31.13:4627 , piece_id=OR633MCN32BCTWPDSEYZJ4BW24VIOCTJQ6CGSGYEAANHSWSUN2OA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.928305240 (590h16m57.799388275s) [noise=true , address=145.79.227.16:4909 , piece_id=PGOL7BRDKIHAHTZ5DLXGWBC3EVHDDXFNAQTXWI6UOIWAFJSPCYLA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.929023799 (590h16m57.800106844s) [noise=true , address=145.79.227.18:4006 , piece_id=KOFSGDCLMN7GUU3H377RHOX4J6AUNP67OEQJJA5S3IBC74GCR62Q, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.929752306 (590h16m57.800835351s) [noise=true , address=77.90.31.12:4590 , piece_id=YVZYH62BGR3TCF5YO4MOYX7I2VQASF4VNLJY4RN27TGGKMV57MDA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.930438052 (590h16m57.801521097s) [noise=true , address=45.91.116.10:4082 , piece_id=SJYDXUTRXIIJWEPAWUZYWUDLJNAO7B4RXVT35BZGHIGUW4DM4IHA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.931053784 (590h16m57.802136829s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXYKHKA6ZMWEUOCHVZBJ6PEA3LVD2XNGAQV77ED3BUJ45DERZYQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.932777928 (590h16m57.803860973s) [noise=true , address=145.79.227.22:4509 , piece_id=PBMUBWNRBBSLXNQZB4GCYISRP22DXRIMOERMR7APVY54AIO3YKUQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.933030889 (590h16m57.804113934s) [noise=true , address=145.79.227.23:4545 , piece_id=BSQTOXAN5R4EMLQCPFCRVALKWWBILRRWP2KMWG7QKRC4UZMXO5CQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.934852207 (590h16m57.805935252s) [noise=true , address=145.79.227.31:4799 , piece_id=JGQEHXZFJQCUXEDUAPSQLSS4XVHNLWA4LNSVXOBV55A4FJU4ZOVA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.935100791 (590h16m57.806183836s) [noise=true , address=145.79.227.52:4446 , piece_id=2632SQSBEVFXLEPZO46DQA6I2TSWHNVQ2X6FJXQFZZB4A2ZOSZSQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.937070271 (590h16m57.808153316s) [noise=true , address=145.79.227.2:4115 , piece_id=VF66MHHCPJ7LVQHJLZD63F652WTIYOZ6Q3LQAIL7JUL6LVPSOTYQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.939269919 (590h16m57.810352964s) [noise=true , address=45.91.116.4:4940 , piece_id=TE223CJK2XIOEHJMHJCLESKMFHXQ54Q35WPQ2WD4G33NU76SNLCA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.941285126 (590h16m57.812368171s) [noise=true , address=145.79.227.17:4971 , piece_id=WEPHUM2MZH357EA6GU6SVSX5ZCHI6WVD23XDBANSK6JE7KBLSSZQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.946017938 (590h16m57.817100984s) [noise=true , address=145.79.227.15:4873 , piece_id=3FGYPBA7RCD3GC472XMFMEXPJXUQXGJJTARQHP4YAR4VW2TYZ3XQ, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:57.948244829 (590h16m57.819327874s) [noise=true , address=77.90.31.10:4517 , piece_id=XRIN7Q24PQRFNT3JWRYZJDCZ2BUM2T26JIGWKGEL64EKLT73YKJA, seg_pos={0 0}, upload=517464]: Uploading piece... Dec 30 05:45:59.552782898 (590h16m59.423865943s) [upload=517464]: Next returned nil segment Dec 30 05:45:59.552799770 (590h16m59.423882805s) [upload=517464]: Waiting for error group managing segments... Dec 30 05:45:59.924238584 (590h16m59.795321619s) [noise=true , address=145.79.227.23:4545 , piece_id=BSQTOXAN5R4EMLQCPFCRVALKWWBILRRWP2KMWG7QKRC4UZMXO5CQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.927785879 (590h16m59.798868924s) [noise=true , address=145.79.227.22:4509 , piece_id=PBMUBWNRBBSLXNQZB4GCYISRP22DXRIMOERMR7APVY54AIO3YKUQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.937793986 (590h16m59.808877021s) [noise=true , address=145.79.227.15:4873 , piece_id=3FGYPBA7RCD3GC472XMFMEXPJXUQXGJJTARQHP4YAR4VW2TYZ3XQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.954720908 (590h16m59.825803943s) [noise=true , address=145.79.227.18:4006 , piece_id=KOFSGDCLMN7GUU3H377RHOX4J6AUNP67OEQJJA5S3IBC74GCR62Q, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.955234606 (590h16m59.826317651s) [noise=true , address=145.79.227.48:4302 , piece_id=3TULKKSZ2XZ5X6XLQ5CONK3XPORAZRPGRNJHN67IU7A6TL3JJ3KA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.956489505 (590h16m59.82757255s) [noise=true , address=145.79.227.16:4909 , piece_id=PGOL7BRDKIHAHTZ5DLXGWBC3EVHDDXFNAQTXWI6UOIWAFJSPCYLA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.960045045 (590h16m59.83112809s) [noise=true , address=145.79.227.17:4971 , piece_id=WEPHUM2MZH357EA6GU6SVSX5ZCHI6WVD23XDBANSK6JE7KBLSSZQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.969394318 (590h16m59.840477383s) [noise=true , address=145.79.227.31:4799 , piece_id=JGQEHXZFJQCUXEDUAPSQLSS4XVHNLWA4LNSVXOBV55A4FJU4ZOVA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:45:59.971143318 (590h16m59.842226364s) [noise=true , address=145.79.227.2:4115 , piece_id=VF66MHHCPJ7LVQHJLZD63F652WTIYOZ6Q3LQAIL7JUL6LVPSOTYQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:46:00.015911375 (590h16m59.88699442s) [noise=true , address=145.79.227.52:4446 , piece_id=2632SQSBEVFXLEPZO46DQA6I2TSWHNVQ2X6FJXQFZZB4A2ZOSZSQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:46:00.104800736 (590h16m59.975883781s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXYKHKA6ZMWEUOCHVZBJ6PEA3LVD2XNGAQV77ED3BUJ45DERZYQ, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:46:00.179880831 (590h17m0.050963876s) [noise=true , address=77.90.31.10:4517 , piece_id=XRIN7Q24PQRFNT3JWRYZJDCZ2BUM2T26JIGWKGEL64EKLT73YKJA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:46:00.335904718 (590h17m0.206987763s) [noise=true , address=77.90.31.13:4627 , piece_id=OR633MCN32BCTWPDSEYZJ4BW24VIOCTJQ6CGSGYEAANHSWSUN2OA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: <nil> Dec 30 05:46:00.335933733 (590h17m0.207016778s) [seg_pos={0 0}, upload=517464]: Segment reached optimal threshold of 13 pieces. Dec 30 05:46:00.336383910 (590h17m0.207466945s) [noise=true , address=77.90.31.12:4590 , piece_id=YVZYH62BGR3TCF5YO4MOYX7I2VQASF4VNLJY4RN27TGGKMV57MDA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:00.336392977 (590h17m0.207476012s) [noise=true , address=77.90.31.12:4590 , piece_id=YVZYH62BGR3TCF5YO4MOYX7I2VQASF4VNLJY4RN27TGGKMV57MDA, seg_pos={0 0}, upload=517464]: Long tail cancellation due to optimal threshold Dec 30 05:46:00.336619098 (590h17m0.207702143s) [noise=true , address=45.91.116.4:4940 , piece_id=TE223CJK2XIOEHJMHJCLESKMFHXQ54Q35WPQ2WD4G33NU76SNLCA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:37800->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:46:00.336626903 (590h17m0.207709938s) [noise=true , address=45.91.116.4:4940 , piece_id=TE223CJK2XIOEHJMHJCLESKMFHXQ54Q35WPQ2WD4G33NU76SNLCA, seg_pos={0 0}, upload=517464]: Long tail cancellation due to optimal threshold Dec 30 05:46:00.336835581 (590h17m0.207918626s) [noise=true , address=45.91.116.10:4082 , piece_id=SJYDXUTRXIIJWEPAWUZYWUDLJNAO7B4RXVT35BZGHIGUW4DM4IHA, seg_pos={0 0}, upload=517464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:44236->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:46:00.336850158 (590h17m0.207933193s) [noise=true , address=45.91.116.10:4082 , piece_id=SJYDXUTRXIIJWEPAWUZYWUDLJNAO7B4RXVT35BZGHIGUW4DM4IHA, seg_pos={0 0}, upload=517464]: Long tail cancellation due to optimal threshold Dec 30 05:46:00.336886487 (590h17m0.207969532s) [seg_pos={0 0}, upload=517464]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:46:00.337793875 (590h17m0.20887692s) [upload=517464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:46:00.337865180 (590h17m0.208948216s) [upload=517464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:46:00.631409189 (590h17m0.502492234s) [upload=517464]: Done uploading segments... Dec 30 05:46:00.631434698 (590h17m0.502517743s) [upload=517464]: Upload finished. err: <nil> Duration: 30.001951227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.X....X........X......................................] duration: 3.608658503s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:46:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608751611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.X....X........X.....................................] duration: 3.541230567s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:45:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541251676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.] duration: 6.094330822s trace: https://jaeger.dev.storj.io/trace/556e39f93e6bd89d FAIL (2025-12-30 05:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ) context deadline exceeded ======LOGS====== Dec 30 05:44:52.394177317 (590h15m52.265260352s) [upload=517448]: Starting upload Dec 30 05:44:52.394306824 (590h15m52.265389869s) [upload=517448]: Done starting upload Dec 30 05:44:52.396927073 (590h15m52.268010118s) [upload=517448]: Uploading segments... Dec 30 05:44:52.397054155 (590h15m52.2681372s) [upload=517448]: Got next segment. Inline: false Dec 30 05:44:52.397097237 (590h15m52.268180272s) [upload=517448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:44:52.397103339 (590h15m52.268186384s) [upload=517448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:44:52.640694809 (590h15m52.511777854s) [seg_pos={0 0}, upload=517448]: Begin upload segment... Dec 30 05:44:52.640757338 (590h15m52.511840373s) [seg_pos={0 0}, upload=517448]: Done begin upload segment. Dec 30 05:44:52.640776705 (590h15m52.51185974s) [noise=true , address=145.79.227.9:4367 , piece_id=2DHYLT2ENFN4TBAVNU2EVVC4OGH52LLYJVHVHMG3DBYIPHYNTLQQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.640838933 (590h15m52.511921968s) [noise=true , address=145.79.227.7:4293 , piece_id=Q7AN2ROBH2J5M2H6WCHIN6XC32FKOV3SYNVB4KPBR6UQDHKKEJVA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.640974411 (590h15m52.512057447s) [noise=true , address=145.79.227.11:4729 , piece_id=KJEPKQYG5GRRWAHZLGE2AIPUVANYXMEFKPYHM3QW737DYHBDPYSQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641257971 (590h15m52.512341016s) [noise=true , address=77.90.31.13:4626 , piece_id=U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641514039 (590h15m52.512597074s) [noise=true , address=145.79.227.46:4603 , piece_id=TLAVZI4MGEXXU3EEYMMJXLZ3DTCAXEEAQODNC4DIQJXAZQJVJTMQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641772391 (590h15m52.512855436s) [noise=true , address=145.79.227.39:4014 , piece_id=SQ2PDKR3BKTNIO7NG3BGYXRYVDQOIBZ35BMWPIZVVFVL4UBP3YIQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.642278705 (590h15m52.51336175s) [noise=true , address=145.79.227.22:4509 , piece_id=22RJEV72WCME7OWQAGFEDPA74DBO5IQKFJYN2MLGOHTUMK6PN3DQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.643719118 (590h15m52.514802153s) [noise=true , address=145.79.227.45:4229 , piece_id=U6GR4V3GCARI7UZXZAEFTZT2B7W6MK4S2OQ5BTSHLEDLG47PONMQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.644608381 (590h15m52.515691416s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.646385625 (590h15m52.51746866s) [noise=true , address=145.79.227.31:4798 , piece_id=7KXMXXX2YVQHWZBIEXDF7GGYQGACBSUY554VLEVKS7ZOQQA7OZWQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.647676894 (590h15m52.518759929s) [noise=true , address=145.79.227.16:4912 , piece_id=SSKUDEFHGKCS3KHNKD2QBQSXJVYWCU37SFZ2PRZD7VVR6LVXZ4IQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.649104732 (590h15m52.520187768s) [noise=true , address=145.79.227.28:4725 , piece_id=66XIKCTZFKXGRZEBRET5FZT27PC5H5RMDUGHBKNPWVRJC4QA2H2A, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.650845337 (590h15m52.521928372s) [noise=true , address=145.79.227.41:4087 , piece_id=3BD6DSFQPIZKLEECHOGS3W3UVVQACKCGJA42OIG7TFGYCDCTGCZA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.652225375 (590h15m52.52330841s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.653560166 (590h15m52.524643201s) [noise=true , address=45.91.116.5:4297 , piece_id=4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.654939193 (590h15m52.526022228s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:54.343472043 (590h15m54.214555078s) [upload=517448]: Next returned nil segment Dec 30 05:44:54.343529222 (590h15m54.214612257s) [upload=517448]: Waiting for error group managing segments... Dec 30 05:44:54.627737213 (590h15m54.498820248s) [noise=true , address=145.79.227.7:4293 , piece_id=Q7AN2ROBH2J5M2H6WCHIN6XC32FKOV3SYNVB4KPBR6UQDHKKEJVA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.628567033 (590h15m54.499650068s) [noise=true , address=145.79.227.39:4014 , piece_id=SQ2PDKR3BKTNIO7NG3BGYXRYVDQOIBZ35BMWPIZVVFVL4UBP3YIQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.638715297 (590h15m54.509798352s) [noise=true , address=145.79.227.16:4912 , piece_id=SSKUDEFHGKCS3KHNKD2QBQSXJVYWCU37SFZ2PRZD7VVR6LVXZ4IQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.639918027 (590h15m54.511001062s) [noise=true , address=145.79.227.45:4229 , piece_id=U6GR4V3GCARI7UZXZAEFTZT2B7W6MK4S2OQ5BTSHLEDLG47PONMQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.641146406 (590h15m54.512229451s) [noise=true , address=145.79.227.28:4725 , piece_id=66XIKCTZFKXGRZEBRET5FZT27PC5H5RMDUGHBKNPWVRJC4QA2H2A, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.642884786 (590h15m54.513967831s) [noise=true , address=145.79.227.11:4729 , piece_id=KJEPKQYG5GRRWAHZLGE2AIPUVANYXMEFKPYHM3QW737DYHBDPYSQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.648881025 (590h15m54.51996407s) [noise=true , address=145.79.227.9:4367 , piece_id=2DHYLT2ENFN4TBAVNU2EVVC4OGH52LLYJVHVHMG3DBYIPHYNTLQQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.651913920 (590h15m54.522996965s) [noise=true , address=145.79.227.46:4603 , piece_id=TLAVZI4MGEXXU3EEYMMJXLZ3DTCAXEEAQODNC4DIQJXAZQJVJTMQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.658897838 (590h15m54.529980883s) [noise=true , address=145.79.227.22:4509 , piece_id=22RJEV72WCME7OWQAGFEDPA74DBO5IQKFJYN2MLGOHTUMK6PN3DQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.668229978 (590h15m54.539313023s) [noise=true , address=145.79.227.31:4798 , piece_id=7KXMXXX2YVQHWZBIEXDF7GGYQGACBSUY554VLEVKS7ZOQQA7OZWQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.730222693 (590h15m54.601305728s) [noise=true , address=145.79.227.41:4087 , piece_id=3BD6DSFQPIZKLEECHOGS3W3UVVQACKCGJA42OIG7TFGYCDCTGCZA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.799982649 (590h15m54.671065694s) [noise=true , address=77.90.31.13:4626 , piece_id=U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:55.392983164 (590h15m55.264066209s) [noise=true , address=45.91.116.5:4297 , piece_id=4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:55.393000907 (590h15m55.264083952s) [seg_pos={0 0}, upload=517448]: Segment reached optimal threshold of 13 pieces. Dec 30 05:44:55.393181230 (590h15m55.264264265s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393187392 (590h15m55.264270427s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393254440 (590h15m55.264337475s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393263257 (590h15m55.264346292s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393443190 (590h15m55.264526225s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393451064 (590h15m55.26453411s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393475261 (590h15m55.264558296s) [seg_pos={0 0}, upload=517448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:44:55.393600539 (590h15m55.264683594s) [upload=517448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:44:55.393607673 (590h15m55.264690728s) [upload=517448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:44:55.678190207 (590h15m55.549273252s) [upload=517448]: Done uploading segments... Dec 30 05:44:55.678209494 (590h15m55.54929253s) [upload=517448]: Upload finished. err: <nil> Duration: 6.094353666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X] duration: 30.003370652s trace: https://jaeger.dev.storj.io/trace/556e39f93e6bd89d FAIL (2025-12-30 05:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ) context deadline exceeded ======LOGS====== Dec 30 05:44:52.394177317 (590h15m52.265260352s) [upload=517448]: Starting upload Dec 30 05:44:52.394306824 (590h15m52.265389869s) [upload=517448]: Done starting upload Dec 30 05:44:52.396927073 (590h15m52.268010118s) [upload=517448]: Uploading segments... Dec 30 05:44:52.397054155 (590h15m52.2681372s) [upload=517448]: Got next segment. Inline: false Dec 30 05:44:52.397097237 (590h15m52.268180272s) [upload=517448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:44:52.397103339 (590h15m52.268186384s) [upload=517448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:44:52.640694809 (590h15m52.511777854s) [seg_pos={0 0}, upload=517448]: Begin upload segment... Dec 30 05:44:52.640757338 (590h15m52.511840373s) [seg_pos={0 0}, upload=517448]: Done begin upload segment. Dec 30 05:44:52.640776705 (590h15m52.51185974s) [noise=true , address=145.79.227.9:4367 , piece_id=2DHYLT2ENFN4TBAVNU2EVVC4OGH52LLYJVHVHMG3DBYIPHYNTLQQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.640838933 (590h15m52.511921968s) [noise=true , address=145.79.227.7:4293 , piece_id=Q7AN2ROBH2J5M2H6WCHIN6XC32FKOV3SYNVB4KPBR6UQDHKKEJVA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.640974411 (590h15m52.512057447s) [noise=true , address=145.79.227.11:4729 , piece_id=KJEPKQYG5GRRWAHZLGE2AIPUVANYXMEFKPYHM3QW737DYHBDPYSQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641257971 (590h15m52.512341016s) [noise=true , address=77.90.31.13:4626 , piece_id=U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641514039 (590h15m52.512597074s) [noise=true , address=145.79.227.46:4603 , piece_id=TLAVZI4MGEXXU3EEYMMJXLZ3DTCAXEEAQODNC4DIQJXAZQJVJTMQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.641772391 (590h15m52.512855436s) [noise=true , address=145.79.227.39:4014 , piece_id=SQ2PDKR3BKTNIO7NG3BGYXRYVDQOIBZ35BMWPIZVVFVL4UBP3YIQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.642278705 (590h15m52.51336175s) [noise=true , address=145.79.227.22:4509 , piece_id=22RJEV72WCME7OWQAGFEDPA74DBO5IQKFJYN2MLGOHTUMK6PN3DQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.643719118 (590h15m52.514802153s) [noise=true , address=145.79.227.45:4229 , piece_id=U6GR4V3GCARI7UZXZAEFTZT2B7W6MK4S2OQ5BTSHLEDLG47PONMQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.644608381 (590h15m52.515691416s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.646385625 (590h15m52.51746866s) [noise=true , address=145.79.227.31:4798 , piece_id=7KXMXXX2YVQHWZBIEXDF7GGYQGACBSUY554VLEVKS7ZOQQA7OZWQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.647676894 (590h15m52.518759929s) [noise=true , address=145.79.227.16:4912 , piece_id=SSKUDEFHGKCS3KHNKD2QBQSXJVYWCU37SFZ2PRZD7VVR6LVXZ4IQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.649104732 (590h15m52.520187768s) [noise=true , address=145.79.227.28:4725 , piece_id=66XIKCTZFKXGRZEBRET5FZT27PC5H5RMDUGHBKNPWVRJC4QA2H2A, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.650845337 (590h15m52.521928372s) [noise=true , address=145.79.227.41:4087 , piece_id=3BD6DSFQPIZKLEECHOGS3W3UVVQACKCGJA42OIG7TFGYCDCTGCZA, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.652225375 (590h15m52.52330841s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.653560166 (590h15m52.524643201s) [noise=true , address=45.91.116.5:4297 , piece_id=4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:52.654939193 (590h15m52.526022228s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Uploading piece... Dec 30 05:44:54.343472043 (590h15m54.214555078s) [upload=517448]: Next returned nil segment Dec 30 05:44:54.343529222 (590h15m54.214612257s) [upload=517448]: Waiting for error group managing segments... Dec 30 05:44:54.627737213 (590h15m54.498820248s) [noise=true , address=145.79.227.7:4293 , piece_id=Q7AN2ROBH2J5M2H6WCHIN6XC32FKOV3SYNVB4KPBR6UQDHKKEJVA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.628567033 (590h15m54.499650068s) [noise=true , address=145.79.227.39:4014 , piece_id=SQ2PDKR3BKTNIO7NG3BGYXRYVDQOIBZ35BMWPIZVVFVL4UBP3YIQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.638715297 (590h15m54.509798352s) [noise=true , address=145.79.227.16:4912 , piece_id=SSKUDEFHGKCS3KHNKD2QBQSXJVYWCU37SFZ2PRZD7VVR6LVXZ4IQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.639918027 (590h15m54.511001062s) [noise=true , address=145.79.227.45:4229 , piece_id=U6GR4V3GCARI7UZXZAEFTZT2B7W6MK4S2OQ5BTSHLEDLG47PONMQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.641146406 (590h15m54.512229451s) [noise=true , address=145.79.227.28:4725 , piece_id=66XIKCTZFKXGRZEBRET5FZT27PC5H5RMDUGHBKNPWVRJC4QA2H2A, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.642884786 (590h15m54.513967831s) [noise=true , address=145.79.227.11:4729 , piece_id=KJEPKQYG5GRRWAHZLGE2AIPUVANYXMEFKPYHM3QW737DYHBDPYSQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.648881025 (590h15m54.51996407s) [noise=true , address=145.79.227.9:4367 , piece_id=2DHYLT2ENFN4TBAVNU2EVVC4OGH52LLYJVHVHMG3DBYIPHYNTLQQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.651913920 (590h15m54.522996965s) [noise=true , address=145.79.227.46:4603 , piece_id=TLAVZI4MGEXXU3EEYMMJXLZ3DTCAXEEAQODNC4DIQJXAZQJVJTMQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.658897838 (590h15m54.529980883s) [noise=true , address=145.79.227.22:4509 , piece_id=22RJEV72WCME7OWQAGFEDPA74DBO5IQKFJYN2MLGOHTUMK6PN3DQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.668229978 (590h15m54.539313023s) [noise=true , address=145.79.227.31:4798 , piece_id=7KXMXXX2YVQHWZBIEXDF7GGYQGACBSUY554VLEVKS7ZOQQA7OZWQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.730222693 (590h15m54.601305728s) [noise=true , address=145.79.227.41:4087 , piece_id=3BD6DSFQPIZKLEECHOGS3W3UVVQACKCGJA42OIG7TFGYCDCTGCZA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:54.799982649 (590h15m54.671065694s) [noise=true , address=77.90.31.13:4626 , piece_id=U4AL47OTXAOTTDPEQ6DKUACI2RBDUWHLNLA52PZJXDV5HDEBS7SA, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:55.392983164 (590h15m55.264066209s) [noise=true , address=45.91.116.5:4297 , piece_id=4M6BV5GQLRGNXKNONBX4Q2JTPL5MWZBEMNVWB7LY6FH7LJT2HTHQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: <nil> Dec 30 05:44:55.393000907 (590h15m55.264083952s) [seg_pos={0 0}, upload=517448]: Segment reached optimal threshold of 13 pieces. Dec 30 05:44:55.393181230 (590h15m55.264264265s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393187392 (590h15m55.264270427s) [noise=true , address=145.79.227.44:4193 , piece_id=ZPE7ASNY4F3PXDBR2F4BUQ5HB5EB5WET7Z2CGC5BHEXXBT44IS2A, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393254440 (590h15m55.264337475s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393263257 (590h15m55.264346292s) [noise=true , address=145.79.227.40:4049 , piece_id=GXQ34X62OKVSZW4BMYBQXC3OU6H2Q7YMT4FCCEOW5J24R6XSTSKQ, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393443190 (590h15m55.264526225s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:44:55.393451064 (590h15m55.26453411s) [noise=true , address=77.90.31.16:4735 , piece_id=TNVCWRVLNICDKHOT2ULFYMUHIVPIARN4ARXJBVVXLV6GFBJMBJPQ, seg_pos={0 0}, upload=517448]: Long tail cancellation due to optimal threshold Dec 30 05:44:55.393475261 (590h15m55.264558296s) [seg_pos={0 0}, upload=517448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:44:55.393600539 (590h15m55.264683594s) [upload=517448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:44:55.393607673 (590h15m55.264690728s) [upload=517448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:44:55.678190207 (590h15m55.549273252s) [upload=517448]: Done uploading segments... Dec 30 05:44:55.678209494 (590h15m55.54929253s) [upload=517448]: Upload finished. err: <nil> Duration: 30.003396241s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.X....X........X....................................] duration: 3.454911802s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:45:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454946258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....] duration: 6.004094527s trace: https://jaeger.dev.storj.io/trace/21b2c73442945f6b FAIL (2025-12-30 05:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA) context deadline exceeded ======LOGS====== Dec 30 05:41:27.222376979 (590h12m27.093460014s) [upload=517399]: Starting upload Dec 30 05:41:27.222431113 (590h12m27.093514158s) [upload=517399]: Done starting upload Dec 30 05:41:27.222613500 (590h12m27.093696535s) [upload=517399]: Uploading segments... Dec 30 05:41:27.222644579 (590h12m27.093727614s) [upload=517399]: Got next segment. Inline: false Dec 30 05:41:27.222667262 (590h12m27.093750298s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:41:27.222670669 (590h12m27.093753714s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:41:27.472977246 (590h12m27.344060281s) [seg_pos={0 0}, upload=517399]: Begin upload segment... Dec 30 05:41:27.473034024 (590h12m27.344117059s) [seg_pos={0 0}, upload=517399]: Done begin upload segment. Dec 30 05:41:27.473049233 (590h12m27.344132268s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473067888 (590h12m27.344150933s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473142360 (590h12m27.344225405s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473685244 (590h12m27.344768289s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475133021 (590h12m27.346216066s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475607144 (590h12m27.346690179s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.476098750 (590h12m27.347181795s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477211709 (590h12m27.348294744s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477638272 (590h12m27.348721307s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.478525781 (590h12m27.349608826s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.483977493 (590h12m27.355060538s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.484176120 (590h12m27.355259155s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.487171544 (590h12m27.358254589s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.488802400 (590h12m27.359885435s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.492304519 (590h12m27.363387554s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.494216900 (590h12m27.365299935s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:29.154300437 (590h12m29.025383472s) [upload=517399]: Next returned nil segment Dec 30 05:41:29.154316136 (590h12m29.025399171s) [upload=517399]: Waiting for error group managing segments... Dec 30 05:41:29.469941813 (590h12m29.341024858s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.486162590 (590h12m29.357245625s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.490572667 (590h12m29.361655702s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496196827 (590h12m29.367279862s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496509873 (590h12m29.367592918s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.563251581 (590h12m29.434334616s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.652748228 (590h12m29.523831273s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.691847810 (590h12m29.562930845s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.776934130 (590h12m29.648017165s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.832720536 (590h12m29.703803571s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.841828009 (590h12m29.712911054s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.911585960 (590h12m29.782669005s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564395562 (590h12m30.435478617s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564416302 (590h12m30.435499337s) [seg_pos={0 0}, upload=517399]: Segment reached optimal threshold of 13 pieces. Dec 30 05:41:30.564622495 (590h12m30.43570554s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564631051 (590h12m30.435714096s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.564931944 (590h12m30.436014999s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564983421 (590h12m30.436066467s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565069987 (590h12m30.436153022s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.565078062 (590h12m30.436161097s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565101757 (590h12m30.436184792s) [seg_pos={0 0}, upload=517399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:41:30.565242155 (590h12m30.4363252s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:41:30.565248447 (590h12m30.436331482s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:41:30.855461553 (590h12m30.726544598s) [upload=517399]: Done uploading segments... Dec 30 05:41:30.855480579 (590h12m30.726563624s) [upload=517399]: Upload finished. err: <nil> Duration: 6.004123552s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.X....X........X...................................] duration: 4.544207609s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:44:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.544261071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X...] duration: 6.103111694s trace: https://jaeger.dev.storj.io/trace/21b2c73442945f6b FAIL (2025-12-30 05:43:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA) context deadline exceeded ======LOGS====== Dec 30 05:41:27.222376979 (590h12m27.093460014s) [upload=517399]: Starting upload Dec 30 05:41:27.222431113 (590h12m27.093514158s) [upload=517399]: Done starting upload Dec 30 05:41:27.222613500 (590h12m27.093696535s) [upload=517399]: Uploading segments... Dec 30 05:41:27.222644579 (590h12m27.093727614s) [upload=517399]: Got next segment. Inline: false Dec 30 05:41:27.222667262 (590h12m27.093750298s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:41:27.222670669 (590h12m27.093753714s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:41:27.472977246 (590h12m27.344060281s) [seg_pos={0 0}, upload=517399]: Begin upload segment... Dec 30 05:41:27.473034024 (590h12m27.344117059s) [seg_pos={0 0}, upload=517399]: Done begin upload segment. Dec 30 05:41:27.473049233 (590h12m27.344132268s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473067888 (590h12m27.344150933s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473142360 (590h12m27.344225405s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473685244 (590h12m27.344768289s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475133021 (590h12m27.346216066s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475607144 (590h12m27.346690179s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.476098750 (590h12m27.347181795s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477211709 (590h12m27.348294744s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477638272 (590h12m27.348721307s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.478525781 (590h12m27.349608826s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.483977493 (590h12m27.355060538s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.484176120 (590h12m27.355259155s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.487171544 (590h12m27.358254589s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.488802400 (590h12m27.359885435s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.492304519 (590h12m27.363387554s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.494216900 (590h12m27.365299935s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:29.154300437 (590h12m29.025383472s) [upload=517399]: Next returned nil segment Dec 30 05:41:29.154316136 (590h12m29.025399171s) [upload=517399]: Waiting for error group managing segments... Dec 30 05:41:29.469941813 (590h12m29.341024858s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.486162590 (590h12m29.357245625s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.490572667 (590h12m29.361655702s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496196827 (590h12m29.367279862s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496509873 (590h12m29.367592918s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.563251581 (590h12m29.434334616s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.652748228 (590h12m29.523831273s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.691847810 (590h12m29.562930845s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.776934130 (590h12m29.648017165s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.832720536 (590h12m29.703803571s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.841828009 (590h12m29.712911054s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.911585960 (590h12m29.782669005s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564395562 (590h12m30.435478617s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564416302 (590h12m30.435499337s) [seg_pos={0 0}, upload=517399]: Segment reached optimal threshold of 13 pieces. Dec 30 05:41:30.564622495 (590h12m30.43570554s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564631051 (590h12m30.435714096s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.564931944 (590h12m30.436014999s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564983421 (590h12m30.436066467s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565069987 (590h12m30.436153022s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.565078062 (590h12m30.436161097s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565101757 (590h12m30.436184792s) [seg_pos={0 0}, upload=517399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:41:30.565242155 (590h12m30.4363252s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:41:30.565248447 (590h12m30.436331482s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:41:30.855461553 (590h12m30.726544598s) [upload=517399]: Done uploading segments... Dec 30 05:41:30.855480579 (590h12m30.726563624s) [upload=517399]: Upload finished. err: <nil> Duration: 6.103139116s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X.X....X........X..................................] duration: 3.558636882s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:42:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559216587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X..] duration: 5.920929386s trace: https://jaeger.dev.storj.io/trace/21b2c73442945f6b FAIL (2025-12-30 05:42:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA) context deadline exceeded ======LOGS====== Dec 30 05:41:27.222376979 (590h12m27.093460014s) [upload=517399]: Starting upload Dec 30 05:41:27.222431113 (590h12m27.093514158s) [upload=517399]: Done starting upload Dec 30 05:41:27.222613500 (590h12m27.093696535s) [upload=517399]: Uploading segments... Dec 30 05:41:27.222644579 (590h12m27.093727614s) [upload=517399]: Got next segment. Inline: false Dec 30 05:41:27.222667262 (590h12m27.093750298s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:41:27.222670669 (590h12m27.093753714s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:41:27.472977246 (590h12m27.344060281s) [seg_pos={0 0}, upload=517399]: Begin upload segment... Dec 30 05:41:27.473034024 (590h12m27.344117059s) [seg_pos={0 0}, upload=517399]: Done begin upload segment. Dec 30 05:41:27.473049233 (590h12m27.344132268s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473067888 (590h12m27.344150933s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473142360 (590h12m27.344225405s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473685244 (590h12m27.344768289s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475133021 (590h12m27.346216066s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475607144 (590h12m27.346690179s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.476098750 (590h12m27.347181795s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477211709 (590h12m27.348294744s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477638272 (590h12m27.348721307s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.478525781 (590h12m27.349608826s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.483977493 (590h12m27.355060538s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.484176120 (590h12m27.355259155s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.487171544 (590h12m27.358254589s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.488802400 (590h12m27.359885435s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.492304519 (590h12m27.363387554s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.494216900 (590h12m27.365299935s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:29.154300437 (590h12m29.025383472s) [upload=517399]: Next returned nil segment Dec 30 05:41:29.154316136 (590h12m29.025399171s) [upload=517399]: Waiting for error group managing segments... Dec 30 05:41:29.469941813 (590h12m29.341024858s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.486162590 (590h12m29.357245625s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.490572667 (590h12m29.361655702s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496196827 (590h12m29.367279862s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496509873 (590h12m29.367592918s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.563251581 (590h12m29.434334616s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.652748228 (590h12m29.523831273s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.691847810 (590h12m29.562930845s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.776934130 (590h12m29.648017165s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.832720536 (590h12m29.703803571s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.841828009 (590h12m29.712911054s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.911585960 (590h12m29.782669005s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564395562 (590h12m30.435478617s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564416302 (590h12m30.435499337s) [seg_pos={0 0}, upload=517399]: Segment reached optimal threshold of 13 pieces. Dec 30 05:41:30.564622495 (590h12m30.43570554s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564631051 (590h12m30.435714096s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.564931944 (590h12m30.436014999s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564983421 (590h12m30.436066467s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565069987 (590h12m30.436153022s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.565078062 (590h12m30.436161097s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565101757 (590h12m30.436184792s) [seg_pos={0 0}, upload=517399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:41:30.565242155 (590h12m30.4363252s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:41:30.565248447 (590h12m30.436331482s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:41:30.855461553 (590h12m30.726544598s) [upload=517399]: Done uploading segments... Dec 30 05:41:30.855480579 (590h12m30.726563624s) [upload=517399]: Upload finished. err: <nil> Duration: 5.920953182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X.X....X........X.................................] duration: 3.469443252s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:42:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46946827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X.] duration: 5.758968644s trace: https://jaeger.dev.storj.io/trace/21b2c73442945f6b FAIL (2025-12-30 05:42:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA) context deadline exceeded ======LOGS====== Dec 30 05:41:27.222376979 (590h12m27.093460014s) [upload=517399]: Starting upload Dec 30 05:41:27.222431113 (590h12m27.093514158s) [upload=517399]: Done starting upload Dec 30 05:41:27.222613500 (590h12m27.093696535s) [upload=517399]: Uploading segments... Dec 30 05:41:27.222644579 (590h12m27.093727614s) [upload=517399]: Got next segment. Inline: false Dec 30 05:41:27.222667262 (590h12m27.093750298s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:41:27.222670669 (590h12m27.093753714s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:41:27.472977246 (590h12m27.344060281s) [seg_pos={0 0}, upload=517399]: Begin upload segment... Dec 30 05:41:27.473034024 (590h12m27.344117059s) [seg_pos={0 0}, upload=517399]: Done begin upload segment. Dec 30 05:41:27.473049233 (590h12m27.344132268s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473067888 (590h12m27.344150933s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473142360 (590h12m27.344225405s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473685244 (590h12m27.344768289s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475133021 (590h12m27.346216066s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475607144 (590h12m27.346690179s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.476098750 (590h12m27.347181795s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477211709 (590h12m27.348294744s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477638272 (590h12m27.348721307s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.478525781 (590h12m27.349608826s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.483977493 (590h12m27.355060538s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.484176120 (590h12m27.355259155s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.487171544 (590h12m27.358254589s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.488802400 (590h12m27.359885435s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.492304519 (590h12m27.363387554s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.494216900 (590h12m27.365299935s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:29.154300437 (590h12m29.025383472s) [upload=517399]: Next returned nil segment Dec 30 05:41:29.154316136 (590h12m29.025399171s) [upload=517399]: Waiting for error group managing segments... Dec 30 05:41:29.469941813 (590h12m29.341024858s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.486162590 (590h12m29.357245625s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.490572667 (590h12m29.361655702s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496196827 (590h12m29.367279862s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496509873 (590h12m29.367592918s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.563251581 (590h12m29.434334616s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.652748228 (590h12m29.523831273s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.691847810 (590h12m29.562930845s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.776934130 (590h12m29.648017165s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.832720536 (590h12m29.703803571s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.841828009 (590h12m29.712911054s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.911585960 (590h12m29.782669005s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564395562 (590h12m30.435478617s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564416302 (590h12m30.435499337s) [seg_pos={0 0}, upload=517399]: Segment reached optimal threshold of 13 pieces. Dec 30 05:41:30.564622495 (590h12m30.43570554s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564631051 (590h12m30.435714096s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.564931944 (590h12m30.436014999s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564983421 (590h12m30.436066467s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565069987 (590h12m30.436153022s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.565078062 (590h12m30.436161097s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565101757 (590h12m30.436184792s) [seg_pos={0 0}, upload=517399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:41:30.565242155 (590h12m30.4363252s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:41:30.565248447 (590h12m30.436331482s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:41:30.855461553 (590h12m30.726544598s) [upload=517399]: Done uploading segments... Dec 30 05:41:30.855480579 (590h12m30.726563624s) [upload=517399]: Upload finished. err: <nil> Duration: 5.758991168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X] duration: 30.003846537s trace: https://jaeger.dev.storj.io/trace/21b2c73442945f6b FAIL (2025-12-30 05:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA) context deadline exceeded ======LOGS====== Dec 30 05:41:27.222376979 (590h12m27.093460014s) [upload=517399]: Starting upload Dec 30 05:41:27.222431113 (590h12m27.093514158s) [upload=517399]: Done starting upload Dec 30 05:41:27.222613500 (590h12m27.093696535s) [upload=517399]: Uploading segments... Dec 30 05:41:27.222644579 (590h12m27.093727614s) [upload=517399]: Got next segment. Inline: false Dec 30 05:41:27.222667262 (590h12m27.093750298s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:41:27.222670669 (590h12m27.093753714s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:41:27.472977246 (590h12m27.344060281s) [seg_pos={0 0}, upload=517399]: Begin upload segment... Dec 30 05:41:27.473034024 (590h12m27.344117059s) [seg_pos={0 0}, upload=517399]: Done begin upload segment. Dec 30 05:41:27.473049233 (590h12m27.344132268s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473067888 (590h12m27.344150933s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473142360 (590h12m27.344225405s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.473685244 (590h12m27.344768289s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475133021 (590h12m27.346216066s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.475607144 (590h12m27.346690179s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.476098750 (590h12m27.347181795s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477211709 (590h12m27.348294744s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.477638272 (590h12m27.348721307s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.478525781 (590h12m27.349608826s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.483977493 (590h12m27.355060538s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.484176120 (590h12m27.355259155s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.487171544 (590h12m27.358254589s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.488802400 (590h12m27.359885435s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.492304519 (590h12m27.363387554s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:27.494216900 (590h12m27.365299935s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Uploading piece... Dec 30 05:41:29.154300437 (590h12m29.025383472s) [upload=517399]: Next returned nil segment Dec 30 05:41:29.154316136 (590h12m29.025399171s) [upload=517399]: Waiting for error group managing segments... Dec 30 05:41:29.469941813 (590h12m29.341024858s) [noise=true , address=145.79.227.26:4653 , piece_id=GB5A4WWBEY5NO4AA2QCTWFR2KWYS6SH56UBNZCJV6XSDMFWJ6M2A, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.486162590 (590h12m29.357245625s) [noise=true , address=145.79.227.49:4339 , piece_id=BVSDTSDXDDU2L3GIE2YNMMAFE37FFEKKHJOK3F5LJ6NKWJHOCDHQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.490572667 (590h12m29.361655702s) [noise=true , address=145.79.227.39:4013 , piece_id=DZTETIAY3EEWPYA6TEPVTXQWFFF3FNXZJ56MZKRIJ4RB3A5WEJHA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496196827 (590h12m29.367279862s) [noise=true , address=145.79.227.8:4331 , piece_id=5FTW5WZ6IW3YEPUTAGKU7FPU74MCGNBCN54BJEU3JD6REHLVNTLA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.496509873 (590h12m29.367592918s) [noise=true , address=145.79.227.5:4221 , piece_id=IZL2R7KX6MQRWQVTH5RB46YLJGTNP3UV5TYJK376FTATINT4TWPA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.563251581 (590h12m29.434334616s) [noise=true , address=145.79.227.16:4909 , piece_id=L4CZAC5RRJIYDP2XKJ34WWO4NRK7RE5PGRUZUWLV2HCAEFAIJYWQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.652748228 (590h12m29.523831273s) [noise=true , address=77.90.31.16:4735 , piece_id=WBGPS4466FCETPRWUN2NWYS2QN4CLRLSBO33V56OS6PEY6NKVEBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.691847810 (590h12m29.562930845s) [noise=true , address=77.90.31.19:4841 , piece_id=TI4IGBZJFKYEERYAQZY6KX52SS2X3RVRVOHYPHC2ZPZMKX4MCZBQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.776934130 (590h12m29.648017165s) [noise=true , address=77.90.31.18:4805 , piece_id=OD75PFIHFIXZT7B5PPVIFWAUL45MBUID2H7WAXHYUBUHF4DX6BJA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.832720536 (590h12m29.703803571s) [noise=true , address=45.91.116.8:4045 , piece_id=HN7DU4IL7NQZRMCIZYH2D4BMZQ3UFPO2X4DI3AEBZKE5Z4SMM7VA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.841828009 (590h12m29.712911054s) [noise=true , address=45.91.116.14:4190 , piece_id=WCZNRIOQETVVBCQ4II4P4TBL6TTUIQAHZ72YO3RKP7USVTSTHURQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:29.911585960 (590h12m29.782669005s) [noise=true , address=45.91.116.12:4226 , piece_id=UPWRPLV4CWHUNCZVYK2ZV2TJPOERFUJK5OOG2RL6MNLXKQT5LMIA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564395562 (590h12m30.435478617s) [noise=true , address=45.91.116.10:4082 , piece_id=QMTUHL7XWIVC4ZNBARHCYROHUEFQXUT4MUQAJVRCCZR3M44FZAMA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: <nil> Dec 30 05:41:30.564416302 (590h12m30.435499337s) [seg_pos={0 0}, upload=517399]: Segment reached optimal threshold of 13 pieces. Dec 30 05:41:30.564622495 (590h12m30.43570554s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564631051 (590h12m30.435714096s) [noise=true , address=45.91.116.13:4155 , piece_id=PT4Y2OX3QNJIV24VSZPVXMP4RJTS7K44ASMLLBA3RNRHBUBYEVMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.564931944 (590h12m30.436014999s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.564983421 (590h12m30.436066467s) [noise=true , address=77.90.31.21:4880 , piece_id=DRZOZEYMSTDJIKKWCWZZJP3OOGWSNL5XZPXISQDDBCI4JPSM6LCA, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565069987 (590h12m30.436153022s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 05:41:30.565078062 (590h12m30.436161097s) [noise=true , address=45.91.116.7:4009 , piece_id=VVGREAQJUXIP6O4KVQUNTGBELJR667PGCCPL2JPEGM4VKKGXAXMQ, seg_pos={0 0}, upload=517399]: Long tail cancellation due to optimal threshold Dec 30 05:41:30.565101757 (590h12m30.436184792s) [seg_pos={0 0}, upload=517399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:41:30.565242155 (590h12m30.4363252s) [upload=517399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:41:30.565248447 (590h12m30.436331482s) [upload=517399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:41:30.855461553 (590h12m30.726544598s) [upload=517399]: Done uploading segments... Dec 30 05:41:30.855480579 (590h12m30.726563624s) [upload=517399]: Upload finished. err: <nil> Duration: 30.003868297s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.X....X........X................................] duration: 3.538878368s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:41:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53890574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..] duration: 5.809394115s trace: https://jaeger.dev.storj.io/trace/42ffab69fba5018b FAIL (2025-12-30 05:40:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ) context deadline exceeded ======LOGS====== ng segments... Dec 30 05:39:45.102888841 (590h10m44.973971886s) [upload=517375]: Got next segment. Inline: false Dec 30 05:39:45.102932604 (590h10m44.974015639s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:39:45.102938194 (590h10m44.974021249s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:39:45.348163870 (590h10m45.219246915s) [seg_pos={0 0}, upload=517375]: Begin upload segment... Dec 30 05:39:45.348220057 (590h10m45.219303092s) [seg_pos={0 0}, upload=517375]: Done begin upload segment. Dec 30 05:39:45.348403125 (590h10m45.21948617s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.348730108 (590h10m45.219813153s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.349937196 (590h10m45.221020231s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.350490720 (590h10m45.221573756s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.351268491 (590h10m45.222351526s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352165770 (590h10m45.223248805s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352250812 (590h10m45.223333847s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353242841 (590h10m45.224325886s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353311732 (590h10m45.224394767s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353834007 (590h10m45.224917042s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.354092710 (590h10m45.225175745s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.355368108 (590h10m45.226451153s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.356013938 (590h10m45.227096973s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358139926 (590h10m45.229222971s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358727776 (590h10m45.229810821s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.359166951 (590h10m45.230249996s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:47.048336184 (590h10m46.919419219s) [upload=517375]: Next returned nil segment Dec 30 05:39:47.048351052 (590h10m46.919434087s) [upload=517375]: Waiting for error group managing segments... Dec 30 05:39:47.333479909 (590h10m47.204562944s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.334066496 (590h10m47.205149541s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.338803396 (590h10m47.209886431s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.339520151 (590h10m47.210603186s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.345669000 (590h10m47.216752035s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.360964666 (590h10m47.232047702s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.362436869 (590h10m47.233519915s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.364464250 (590h10m47.235547295s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.365841823 (590h10m47.236924878s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.366388123 (590h10m47.237471158s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.454441143 (590h10m47.325524188s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.576624212 (590h10m47.447707247s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735300441 (590h10m47.606383486s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735324726 (590h10m47.606407762s) [seg_pos={0 0}, upload=517375]: Segment reached optimal threshold of 13 pieces. Dec 30 05:39:47.735545277 (590h10m47.606628312s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection Dec 30 05:39:47.735560355 (590h10m47.60664339s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735644186 (590h10m47.606727221s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection Dec 30 05:39:47.735667509 (590h10m47.606750544s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735752111 (590h10m47.606835146s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 05:39:47.735760407 (590h10m47.606843442s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735785805 (590h10m47.60686885s) [seg_pos={0 0}, upload=517375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:39:47.735890464 (590h10m47.60697351s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:39:47.735898760 (590h10m47.606981795s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:39:48.027973365 (590h10m47.8990564s) [upload=517375]: Done uploading segments... Dec 30 05:39:48.027994314 (590h10m47.89907736s) [upload=517375]: Upload finished. err: <nil> Duration: 5.809419464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X.] duration: 5.739694684s trace: https://jaeger.dev.storj.io/trace/42ffab69fba5018b FAIL (2025-12-30 05:40:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ) context deadline exceeded ======LOGS====== ng segments... Dec 30 05:39:45.102888841 (590h10m44.973971886s) [upload=517375]: Got next segment. Inline: false Dec 30 05:39:45.102932604 (590h10m44.974015639s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:39:45.102938194 (590h10m44.974021249s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:39:45.348163870 (590h10m45.219246915s) [seg_pos={0 0}, upload=517375]: Begin upload segment... Dec 30 05:39:45.348220057 (590h10m45.219303092s) [seg_pos={0 0}, upload=517375]: Done begin upload segment. Dec 30 05:39:45.348403125 (590h10m45.21948617s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.348730108 (590h10m45.219813153s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.349937196 (590h10m45.221020231s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.350490720 (590h10m45.221573756s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.351268491 (590h10m45.222351526s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352165770 (590h10m45.223248805s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352250812 (590h10m45.223333847s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353242841 (590h10m45.224325886s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353311732 (590h10m45.224394767s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353834007 (590h10m45.224917042s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.354092710 (590h10m45.225175745s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.355368108 (590h10m45.226451153s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.356013938 (590h10m45.227096973s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358139926 (590h10m45.229222971s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358727776 (590h10m45.229810821s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.359166951 (590h10m45.230249996s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:47.048336184 (590h10m46.919419219s) [upload=517375]: Next returned nil segment Dec 30 05:39:47.048351052 (590h10m46.919434087s) [upload=517375]: Waiting for error group managing segments... Dec 30 05:39:47.333479909 (590h10m47.204562944s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.334066496 (590h10m47.205149541s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.338803396 (590h10m47.209886431s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.339520151 (590h10m47.210603186s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.345669000 (590h10m47.216752035s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.360964666 (590h10m47.232047702s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.362436869 (590h10m47.233519915s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.364464250 (590h10m47.235547295s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.365841823 (590h10m47.236924878s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.366388123 (590h10m47.237471158s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.454441143 (590h10m47.325524188s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.576624212 (590h10m47.447707247s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735300441 (590h10m47.606383486s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735324726 (590h10m47.606407762s) [seg_pos={0 0}, upload=517375]: Segment reached optimal threshold of 13 pieces. Dec 30 05:39:47.735545277 (590h10m47.606628312s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection Dec 30 05:39:47.735560355 (590h10m47.60664339s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735644186 (590h10m47.606727221s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection Dec 30 05:39:47.735667509 (590h10m47.606750544s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735752111 (590h10m47.606835146s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 05:39:47.735760407 (590h10m47.606843442s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735785805 (590h10m47.60686885s) [seg_pos={0 0}, upload=517375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:39:47.735890464 (590h10m47.60697351s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:39:47.735898760 (590h10m47.606981795s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:39:48.027973365 (590h10m47.8990564s) [upload=517375]: Done uploading segments... Dec 30 05:39:48.027994314 (590h10m47.89907736s) [upload=517375]: Upload finished. err: <nil> Duration: 5.739723148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.X....X........X...............................] duration: 3.71172641s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:40:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.711754274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X] duration: 30.002880821s trace: https://jaeger.dev.storj.io/trace/42ffab69fba5018b FAIL (2025-12-30 05:40:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ) context deadline exceeded ======LOGS====== ng segments... Dec 30 05:39:45.102888841 (590h10m44.973971886s) [upload=517375]: Got next segment. Inline: false Dec 30 05:39:45.102932604 (590h10m44.974015639s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:39:45.102938194 (590h10m44.974021249s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:39:45.348163870 (590h10m45.219246915s) [seg_pos={0 0}, upload=517375]: Begin upload segment... Dec 30 05:39:45.348220057 (590h10m45.219303092s) [seg_pos={0 0}, upload=517375]: Done begin upload segment. Dec 30 05:39:45.348403125 (590h10m45.21948617s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.348730108 (590h10m45.219813153s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.349937196 (590h10m45.221020231s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.350490720 (590h10m45.221573756s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.351268491 (590h10m45.222351526s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352165770 (590h10m45.223248805s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.352250812 (590h10m45.223333847s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353242841 (590h10m45.224325886s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353311732 (590h10m45.224394767s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.353834007 (590h10m45.224917042s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.354092710 (590h10m45.225175745s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.355368108 (590h10m45.226451153s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.356013938 (590h10m45.227096973s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358139926 (590h10m45.229222971s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.358727776 (590h10m45.229810821s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:45.359166951 (590h10m45.230249996s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Uploading piece... Dec 30 05:39:47.048336184 (590h10m46.919419219s) [upload=517375]: Next returned nil segment Dec 30 05:39:47.048351052 (590h10m46.919434087s) [upload=517375]: Waiting for error group managing segments... Dec 30 05:39:47.333479909 (590h10m47.204562944s) [noise=true , address=145.79.227.5:4221 , piece_id=OH54DS5UJEOJPP2XSS5YEADX5BMF43II5XDVAVINFGSZVXYDAQMQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.334066496 (590h10m47.205149541s) [noise=true , address=145.79.227.48:4302 , piece_id=AN2MLFZHSXVJKOQTYHJXMTMBVZDW6R3L4GX4CL32BWCFTYMMGABA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.338803396 (590h10m47.209886431s) [noise=true , address=145.79.227.13:4802 , piece_id=R4KPOUFLJCTKYNAFQ3NRU5L5GFVJIQVDUFBOX57KQJS3P3PC3SYQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.339520151 (590h10m47.210603186s) [noise=true , address=145.79.227.51:4410 , piece_id=CU4GJJAZ5GJ43JAOX7RM63CTZOBYI5S2ATDHCDFZ344RIHANUXDA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.345669000 (590h10m47.216752035s) [noise=true , address=145.79.227.20:4439 , piece_id=RK5IDF4YLQK5GG4TOIT627EIBZWOMNEMUKV4WIKBURTSPQ6VUWGA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.360964666 (590h10m47.232047702s) [noise=true , address=145.79.227.14:4838 , piece_id=DHEPFXMF4G2HMPMF3PPJO2LUVZVOTFRTEPQG2RDKMG6FSKZMXACQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.362436869 (590h10m47.233519915s) [noise=true , address=145.79.227.52:4445 , piece_id=RY2Y2EGV4L3IYNLXHX3KP6NFRBFHVXC24KHZKOJSQ65ILMS66BJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.364464250 (590h10m47.235547295s) [noise=true , address=145.79.227.6:4257 , piece_id=SJMQ7VDLG4SQOROK5DPA74PCGRQIUZ3GFQ4OLSDXFLTVEN4T45HA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.365841823 (590h10m47.236924878s) [noise=true , address=145.79.227.38:4977 , piece_id=72YJPZUEM6KOMNWLJHLIPVM5VQEZQK3XZVPV4TC37EGFOWV7MTLQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.366388123 (590h10m47.237471158s) [noise=true , address=145.79.227.2:4114 , piece_id=GYSUEZSHCZGKUQQB5QR6UISJOGAG3VLEL6M3KHXE7KX4XM2A6R2Q, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.454441143 (590h10m47.325524188s) [noise=true , address=145.79.227.16:4912 , piece_id=WMTXI67VWUFO3NSXF5YKEMPSDHQ5NCIJKUKQV4ILSRD4RDOOMDRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.576624212 (590h10m47.447707247s) [noise=true , address=77.90.31.15:4700 , piece_id=BB5YUQIOANQ2NGIPJBIIHBEDA4B54H5OQMVUJ5EZNA4PGQQZAWIQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735300441 (590h10m47.606383486s) [noise=true , address=45.91.116.6:4975 , piece_id=ZW72S5H4O3K2IVNHJM3F3FSFIY53TOTX6SDU4XO5XBZEKUDO2PFQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: <nil> Dec 30 05:39:47.735324726 (590h10m47.606407762s) [seg_pos={0 0}, upload=517375]: Segment reached optimal threshold of 13 pieces. Dec 30 05:39:47.735545277 (590h10m47.606628312s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53680->77.90.31.19:4844: use of closed network connection Dec 30 05:39:47.735560355 (590h10m47.60664339s) [noise=true , address=77.90.31.19:4844 , piece_id=CYA7CVEEFIHYAG25YSSSA6WT2W3FZKEVPYTDH2O2V2LKRNIS5YJQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735644186 (590h10m47.606727221s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50938->45.91.116.5:4299: use of closed network connection Dec 30 05:39:47.735667509 (590h10m47.606750544s) [noise=true , address=45.91.116.5:4299 , piece_id=LZHS2YK4ZSPAGSUG3BBGOWYTR7CKH4GPU5Y54GSXEO2C27C6SYRA, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735752111 (590h10m47.606835146s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 05:39:47.735760407 (590h10m47.606843442s) [noise=true , address=145.79.227.35:4482 , piece_id=TLOCP6GD6EZK24R4JU4XUIY22CNL2RZFYLHYLUDHOQVZ6PPUZ4JQ, seg_pos={0 0}, upload=517375]: Long tail cancellation due to optimal threshold Dec 30 05:39:47.735785805 (590h10m47.60686885s) [seg_pos={0 0}, upload=517375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:39:47.735890464 (590h10m47.60697351s) [upload=517375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:39:47.735898760 (590h10m47.606981795s) [upload=517375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:39:48.027973365 (590h10m47.8990564s) [upload=517375]: Done uploading segments... Dec 30 05:39:48.027994314 (590h10m47.89907736s) [upload=517375]: Upload finished. err: <nil> Duration: 30.002905699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 44.525869154s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:39:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 44.526063935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..] duration: 5.758021269s trace: https://jaeger.dev.storj.io/trace/3498df19ebc6c8fa FAIL (2025-12-30 05:39:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:37:44.578481934 (590h8m44.449564979s) [upload=517345]: Starting upload Dec 30 05:37:44.578573237 (590h8m44.449656272s) [upload=517345]: Done starting upload Dec 30 05:37:44.578621539 (590h8m44.449704604s) [upload=517345]: Uploading segments... Dec 30 05:37:44.579124136 (590h8m44.450207191s) [upload=517345]: Got next segment. Inline: false Dec 30 05:37:44.579177688 (590h8m44.450260723s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:37:44.579185844 (590h8m44.450268879s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:37:44.827022370 (590h8m44.698105415s) [seg_pos={0 0}, upload=517345]: Begin upload segment... Dec 30 05:37:44.827087364 (590h8m44.698170439s) [seg_pos={0 0}, upload=517345]: Done begin upload segment. Dec 30 05:37:44.827137378 (590h8m44.698220413s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827167706 (590h8m44.698250751s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827637141 (590h8m44.698720186s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828193350 (590h8m44.699276385s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828397849 (590h8m44.699480894s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828737495 (590h8m44.69982054s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829032407 (590h8m44.700115452s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829164239 (590h8m44.700247284s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829300939 (590h8m44.700383974s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829590280 (590h8m44.700673315s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829602242 (590h8m44.700685277s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829944404 (590h8m44.701027449s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830344516 (590h8m44.701427561s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830488901 (590h8m44.701571946s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831137285 (590h8m44.70222033s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831654981 (590h8m44.702738016s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:46.436595366 (590h8m46.307678411s) [upload=517345]: Next returned nil segment Dec 30 05:37:46.436615073 (590h8m46.307698118s) [upload=517345]: Waiting for error group managing segments... Dec 30 05:37:46.819679148 (590h8m46.690762193s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.820436279 (590h8m46.691519315s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.821447295 (590h8m46.69253034s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.829576905 (590h8m46.70065995s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.845983287 (590h8m46.717066322s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.902036541 (590h8m46.773119586s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.903742159 (590h8m46.774825194s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.998051472 (590h8m46.869134507s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.069085123 (590h8m46.940168158s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.124944039 (590h8m46.996027084s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.139138547 (590h8m47.010221592s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.219442511 (590h8m47.090525556s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239103190 (590h8m47.110186225s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239121083 (590h8m47.110204128s) [seg_pos={0 0}, upload=517345]: Segment reached optimal threshold of 13 pieces. Dec 30 05:37:47.239291187 (590h8m47.110374222s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239297008 (590h8m47.110380053s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239442275 (590h8m47.11052531s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239448036 (590h8m47.110531071s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239458576 (590h8m47.110541621s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection Dec 30 05:37:47.239468855 (590h8m47.1105519s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239488533 (590h8m47.110571578s) [seg_pos={0 0}, upload=517345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:37:47.239566632 (590h8m47.110649667s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:37:47.239573264 (590h8m47.110656309s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:37:47.526954461 (590h8m47.398037506s) [upload=517345]: Done uploading segments... Dec 30 05:37:47.526976744 (590h8m47.398059789s) [upload=517345]: Upload finished. err: <nil> Duration: 5.758051858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X............X.X....X........X..............................] duration: 3.466252889s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:39:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466278567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X.] duration: 5.70709014s trace: https://jaeger.dev.storj.io/trace/3498df19ebc6c8fa FAIL (2025-12-30 05:38:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:37:44.578481934 (590h8m44.449564979s) [upload=517345]: Starting upload Dec 30 05:37:44.578573237 (590h8m44.449656272s) [upload=517345]: Done starting upload Dec 30 05:37:44.578621539 (590h8m44.449704604s) [upload=517345]: Uploading segments... Dec 30 05:37:44.579124136 (590h8m44.450207191s) [upload=517345]: Got next segment. Inline: false Dec 30 05:37:44.579177688 (590h8m44.450260723s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:37:44.579185844 (590h8m44.450268879s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:37:44.827022370 (590h8m44.698105415s) [seg_pos={0 0}, upload=517345]: Begin upload segment... Dec 30 05:37:44.827087364 (590h8m44.698170439s) [seg_pos={0 0}, upload=517345]: Done begin upload segment. Dec 30 05:37:44.827137378 (590h8m44.698220413s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827167706 (590h8m44.698250751s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827637141 (590h8m44.698720186s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828193350 (590h8m44.699276385s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828397849 (590h8m44.699480894s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828737495 (590h8m44.69982054s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829032407 (590h8m44.700115452s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829164239 (590h8m44.700247284s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829300939 (590h8m44.700383974s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829590280 (590h8m44.700673315s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829602242 (590h8m44.700685277s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829944404 (590h8m44.701027449s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830344516 (590h8m44.701427561s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830488901 (590h8m44.701571946s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831137285 (590h8m44.70222033s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831654981 (590h8m44.702738016s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:46.436595366 (590h8m46.307678411s) [upload=517345]: Next returned nil segment Dec 30 05:37:46.436615073 (590h8m46.307698118s) [upload=517345]: Waiting for error group managing segments... Dec 30 05:37:46.819679148 (590h8m46.690762193s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.820436279 (590h8m46.691519315s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.821447295 (590h8m46.69253034s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.829576905 (590h8m46.70065995s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.845983287 (590h8m46.717066322s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.902036541 (590h8m46.773119586s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.903742159 (590h8m46.774825194s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.998051472 (590h8m46.869134507s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.069085123 (590h8m46.940168158s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.124944039 (590h8m46.996027084s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.139138547 (590h8m47.010221592s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.219442511 (590h8m47.090525556s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239103190 (590h8m47.110186225s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239121083 (590h8m47.110204128s) [seg_pos={0 0}, upload=517345]: Segment reached optimal threshold of 13 pieces. Dec 30 05:37:47.239291187 (590h8m47.110374222s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239297008 (590h8m47.110380053s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239442275 (590h8m47.11052531s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239448036 (590h8m47.110531071s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239458576 (590h8m47.110541621s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection Dec 30 05:37:47.239468855 (590h8m47.1105519s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239488533 (590h8m47.110571578s) [seg_pos={0 0}, upload=517345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:37:47.239566632 (590h8m47.110649667s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:37:47.239573264 (590h8m47.110656309s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:37:47.526954461 (590h8m47.398037506s) [upload=517345]: Done uploading segments... Dec 30 05:37:47.526976744 (590h8m47.398059789s) [upload=517345]: Upload finished. err: <nil> Duration: 5.707115978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 32.979358119s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:38:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 32.979380012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X] duration: 30.002296447s trace: https://jaeger.dev.storj.io/trace/3498df19ebc6c8fa FAIL (2025-12-30 05:38:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:37:44.578481934 (590h8m44.449564979s) [upload=517345]: Starting upload Dec 30 05:37:44.578573237 (590h8m44.449656272s) [upload=517345]: Done starting upload Dec 30 05:37:44.578621539 (590h8m44.449704604s) [upload=517345]: Uploading segments... Dec 30 05:37:44.579124136 (590h8m44.450207191s) [upload=517345]: Got next segment. Inline: false Dec 30 05:37:44.579177688 (590h8m44.450260723s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:37:44.579185844 (590h8m44.450268879s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:37:44.827022370 (590h8m44.698105415s) [seg_pos={0 0}, upload=517345]: Begin upload segment... Dec 30 05:37:44.827087364 (590h8m44.698170439s) [seg_pos={0 0}, upload=517345]: Done begin upload segment. Dec 30 05:37:44.827137378 (590h8m44.698220413s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827167706 (590h8m44.698250751s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.827637141 (590h8m44.698720186s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828193350 (590h8m44.699276385s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828397849 (590h8m44.699480894s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.828737495 (590h8m44.69982054s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829032407 (590h8m44.700115452s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829164239 (590h8m44.700247284s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829300939 (590h8m44.700383974s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829590280 (590h8m44.700673315s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829602242 (590h8m44.700685277s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.829944404 (590h8m44.701027449s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830344516 (590h8m44.701427561s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.830488901 (590h8m44.701571946s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831137285 (590h8m44.70222033s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:44.831654981 (590h8m44.702738016s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Uploading piece... Dec 30 05:37:46.436595366 (590h8m46.307678411s) [upload=517345]: Next returned nil segment Dec 30 05:37:46.436615073 (590h8m46.307698118s) [upload=517345]: Waiting for error group managing segments... Dec 30 05:37:46.819679148 (590h8m46.690762193s) [noise=true , address=145.79.227.36:4517 , piece_id=YV3SBGWXPQJXM4K7QP7IQE5D5QGXNMRHINL7ALTRKOO7UASNI2GA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.820436279 (590h8m46.691519315s) [noise=true , address=145.79.227.5:4221 , piece_id=5CI4FM5K2GUB77JI7LLMT5VI2MJQYX242STSYO3EFOWH2AI7U6NQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.821447295 (590h8m46.69253034s) [noise=true , address=145.79.227.28:4727 , piece_id=ROJGHFR6DSZPZRUGBMYTTPHEATSGANYCPV6KDDEAFEL7LCNRC6BQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.829576905 (590h8m46.70065995s) [noise=true , address=145.79.227.3:4151 , piece_id=E3YDVOTO5KZSE6S3LEJCJ72SBJ4SPCHT4SGKPVLUL7EH7QY74QNA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.845983287 (590h8m46.717066322s) [noise=true , address=145.79.227.26:4654 , piece_id=JYO6SXEZZVAEH6OKV5O7TZKBRGICMV2VFCOXPWBCJSHUZHEZMZKA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.902036541 (590h8m46.773119586s) [noise=true , address=145.79.227.51:4411 , piece_id=VZI4BY33HNEXU2OCXTTF66KEVRBFODHNE5QUGAQFFFZLF5HBBWMA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.903742159 (590h8m46.774825194s) [noise=true , address=145.79.227.7:4294 , piece_id=BFZ2MGWU4QBLMONGXMF3CJW33CE2MWTG7JW6ZQNLOJ7SLQPPSZOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:46.998051472 (590h8m46.869134507s) [noise=true , address=77.90.31.11:4556 , piece_id=WTM3KABESZE6C4QB2LRN3CFZPK5IUJDAU6ZHXPNIFL2CO3EOYIGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.069085123 (590h8m46.940168158s) [noise=true , address=145.79.227.40:4050 , piece_id=GTY2OIGLM4BABURJ36X4HUM4CRRJ3ROIPE5FWMFRVNTDS7P5W54Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.124944039 (590h8m46.996027084s) [noise=true , address=77.90.31.21:4878 , piece_id=56WJT7GEX4ZRESYW2SKGFZ5JJICS5H7PR6IF63V3REHDSDX4PU3Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.139138547 (590h8m47.010221592s) [noise=true , address=77.90.31.14:4661 , piece_id=45IMJ3OMQORMDX5CONQXDQKZZPW5UZTTAYM6D6KBNUG5GGT3AMOQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.219442511 (590h8m47.090525556s) [noise=true , address=45.91.116.8:4048 , piece_id=R4SJGN5MIT3MQUH45RAAGKQHZGR5KXVXRHSZOOPBMBQSB4D3KS6Q, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239103190 (590h8m47.110186225s) [noise=true , address=45.91.116.10:4082 , piece_id=VCU6D3LJBJP2BVPFK2CX6SWJSNLE7KSYMJTM6FG7ARZ3YFHQNTRQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: <nil> Dec 30 05:37:47.239121083 (590h8m47.110204128s) [seg_pos={0 0}, upload=517345]: Segment reached optimal threshold of 13 pieces. Dec 30 05:37:47.239291187 (590h8m47.110374222s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239297008 (590h8m47.110380053s) [noise=true , address=145.79.227.35:4483 , piece_id=TACBEU6M5F6KKCCICQHXJLMFJQJLOLJ5IX734ELW2TVL772PPCCQ, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239442275 (590h8m47.11052531s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 05:37:47.239448036 (590h8m47.110531071s) [noise=true , address=145.79.227.44:4194 , piece_id=XYM7OCTGUQZKDRD5TP5XGO7T4P6YTIK4PD6F6ONURPA2CE2UTBGA, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239458576 (590h8m47.110541621s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58588->45.91.116.7:4009: use of closed network connection Dec 30 05:37:47.239468855 (590h8m47.1105519s) [noise=true , address=45.91.116.7:4009 , piece_id=TAQPZH6IGEDPVMMIXD5DA5M2CDHQX7XVW3VUT5QVNBLCNW5EZC3A, seg_pos={0 0}, upload=517345]: Long tail cancellation due to optimal threshold Dec 30 05:37:47.239488533 (590h8m47.110571578s) [seg_pos={0 0}, upload=517345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:37:47.239566632 (590h8m47.110649667s) [upload=517345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:37:47.239573264 (590h8m47.110656309s) [upload=517345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:37:47.526954461 (590h8m47.398037506s) [upload=517345]: Done uploading segments... Dec 30 05:37:47.526976744 (590h8m47.398059789s) [upload=517345]: Upload finished. err: <nil> Duration: 30.002384654s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X............X.X....X........X.............................] duration: 3.49329981s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:38:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493381456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 38.027979024s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:37:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 38.027998962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X............X.X....X........X............................] duration: 3.440206781s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:37:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440229693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 48.489831798s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:37:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 48.489858419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X.......] duration: 5.756252323s trace: https://jaeger.dev.storj.io/trace/62601e16b7e3acab FAIL (2025-12-30 05:36:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:31:25.686772372 (590h2m25.557855417s) [upload=517249]: Starting upload Dec 30 05:31:25.686864357 (590h2m25.557947402s) [upload=517249]: Done starting upload Dec 30 05:31:25.686967083 (590h2m25.558050128s) [upload=517249]: Uploading segments... Dec 30 05:31:25.687144340 (590h2m25.558227395s) [upload=517249]: Got next segment. Inline: false Dec 30 05:31:25.687194385 (590h2m25.55827743s) [upload=517249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:31:25.687199926 (590h2m25.558282972s) [upload=517249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:31:25.934194068 (590h2m25.805277123s) [seg_pos={0 0}, upload=517249]: Begin upload segment... Dec 30 05:31:25.934325408 (590h2m25.805408443s) [seg_pos={0 0}, upload=517249]: Done begin upload segment. Dec 30 05:31:25.934365684 (590h2m25.80544875s) [noise=true , address=77.90.31.11:4553 , piece_id=EZYGMFLFH4JYQRGYKBBIZW7YOHMXR2HHUMJPQVSECEK6I5CLDPMA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.934514447 (590h2m25.805597493s) [noise=true , address=145.79.227.46:4601 , piece_id=GYKUVKJU7PFON6GQNNEBURILJURMNV4AMWPD4BSZELBPKHN3WUBQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935171739 (590h2m25.806254784s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935430823 (590h2m25.806513868s) [noise=true , address=145.79.227.4:4185 , piece_id=CHL4TSLOFWIYSI4YA23PCCCZMQSW4XKVEXZMQCBLFR22KQATESZA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935641374 (590h2m25.806724409s) [noise=true , address=145.79.227.2:4114 , piece_id=NA7CRCYCYTU4C4EHV6ED6LHUWQRTB2IPIVPIURK4M3GHXYUJ4RHA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936078195 (590h2m25.80716123s) [noise=true , address=145.79.227.10:4402 , piece_id=ZIY5G3WUK3G6MJQYJGHTA25ZVVAM76W6O5O6LH3ZHBNAMKWEJXKQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936084498 (590h2m25.807167543s) [noise=true , address=145.79.227.3:4150 , piece_id=3YKCLWHRPVWF52O7L3Q3ID5Z7ZTIPNRLYYGWKQYSGF5BJB6SOWCQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936661626 (590h2m25.807744671s) [noise=true , address=45.91.116.14:4191 , piece_id=FOCBVJUUTKCRM74DS5RHRUS47W2Z2Y5A4OQSOGTJ6EC7OB4V6YGA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936682927 (590h2m25.807765972s) [noise=true , address=145.79.227.26:4653 , piece_id=NA5MU225IFHVDEDZEEUFOBOWFBK7EB25XP3WNOWOZVEL3LOGFYRA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937168342 (590h2m25.808251387s) [noise=true , address=145.79.227.28:4725 , piece_id=ZWA5DQN6YAG6VYMLVIONKNCIAWZ6GT7URVPM2QVAO4INRSZLUKAQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937452473 (590h2m25.808535538s) [noise=true , address=145.79.227.18:4006 , piece_id=DHUB7DOY225PR7T5BFMCDDE5JD5FS7O5TZMDS5IWNR34JMD3JIEA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937758335 (590h2m25.80884138s) [noise=true , address=145.79.227.36:4517 , piece_id=VHMRDJJXAPLM6X5LEXYMAB6D2EWAI7EVITX4RDPM6HQ5HEW5IVMQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.938376161 (590h2m25.809459206s) [noise=true , address=45.91.116.6:4973 , piece_id=J4E3DDN7XFTQ3T3OQ3VOP7LVCGOFH4QKXIUXCRIURR7XIR5KVUXQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.938383937 (590h2m25.809466972s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.939067167 (590h2m25.810150212s) [noise=true , address=145.79.227.50:4374 , piece_id=RXGBIGU5X3VPDQKTJCBGHG62LZ5RWLECB3QIT5FUL73U3FO5R77Q, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.939756139 (590h2m25.810839194s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:27.639130403 (590h2m27.510213438s) [upload=517249]: Next returned nil segment Dec 30 05:31:27.639148108 (590h2m27.510231153s) [upload=517249]: Waiting for error group managing segments... Dec 30 05:31:27.918866023 (590h2m27.789949078s) [noise=true , address=145.79.227.36:4517 , piece_id=VHMRDJJXAPLM6X5LEXYMAB6D2EWAI7EVITX4RDPM6HQ5HEW5IVMQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.919204248 (590h2m27.790287283s) [noise=true , address=145.79.227.4:4185 , piece_id=CHL4TSLOFWIYSI4YA23PCCCZMQSW4XKVEXZMQCBLFR22KQATESZA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.923402212 (590h2m27.794485257s) [noise=true , address=145.79.227.26:4653 , piece_id=NA5MU225IFHVDEDZEEUFOBOWFBK7EB25XP3WNOWOZVEL3LOGFYRA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.930735455 (590h2m27.8018185s) [noise=true , address=145.79.227.28:4725 , piece_id=ZWA5DQN6YAG6VYMLVIONKNCIAWZ6GT7URVPM2QVAO4INRSZLUKAQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.938066956 (590h2m27.809149991s) [noise=true , address=145.79.227.18:4006 , piece_id=DHUB7DOY225PR7T5BFMCDDE5JD5FS7O5TZMDS5IWNR34JMD3JIEA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.942031345 (590h2m27.8131144s) [noise=true , address=145.79.227.50:4374 , piece_id=RXGBIGU5X3VPDQKTJCBGHG62LZ5RWLECB3QIT5FUL73U3FO5R77Q, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.942575931 (590h2m27.813658966s) [noise=true , address=145.79.227.3:4150 , piece_id=3YKCLWHRPVWF52O7L3Q3ID5Z7ZTIPNRLYYGWKQYSGF5BJB6SOWCQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.943193638 (590h2m27.814276683s) [noise=true , address=145.79.227.46:4601 , piece_id=GYKUVKJU7PFON6GQNNEBURILJURMNV4AMWPD4BSZELBPKHN3WUBQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.963317117 (590h2m27.834400152s) [noise=true , address=145.79.227.2:4114 , piece_id=NA7CRCYCYTU4C4EHV6ED6LHUWQRTB2IPIVPIURK4M3GHXYUJ4RHA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.995157196 (590h2m27.866240231s) [noise=true , address=145.79.227.10:4402 , piece_id=ZIY5G3WUK3G6MJQYJGHTA25ZVVAM76W6O5O6LH3ZHBNAMKWEJXKQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.227384079 (590h2m28.098467114s) [noise=true , address=77.90.31.11:4553 , piece_id=EZYGMFLFH4JYQRGYKBBIZW7YOHMXR2HHUMJPQVSECEK6I5CLDPMA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.237871599 (590h2m28.108954634s) [noise=true , address=45.91.116.6:4973 , piece_id=J4E3DDN7XFTQ3T3OQ3VOP7LVCGOFH4QKXIUXCRIURR7XIR5KVUXQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.380368961 (590h2m28.251451996s) [noise=true , address=45.91.116.14:4191 , piece_id=FOCBVJUUTKCRM74DS5RHRUS47W2Z2Y5A4OQSOGTJ6EC7OB4V6YGA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.380391995 (590h2m28.25147504s) [seg_pos={0 0}, upload=517249]: Segment reached optimal threshold of 13 pieces. Dec 30 05:31:28.380585093 (590h2m28.251668138s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 05:31:28.380594079 (590h2m28.251677134s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380645388 (590h2m28.251728423s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection Dec 30 05:31:28.380656158 (590h2m28.251739223s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380848304 (590h2m28.251931339s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 05:31:28.380858403 (590h2m28.251941438s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380879233 (590h2m28.251962278s) [seg_pos={0 0}, upload=517249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:31:28.380965317 (590h2m28.252048362s) [upload=517249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:31:28.380986517 (590h2m28.252069562s) [upload=517249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:31:28.668586000 (590h2m28.539669035s) [upload=517249]: Done uploading segments... Dec 30 05:31:28.668609435 (590h2m28.53969247s) [upload=517249]: Upload finished. err: <nil> Duration: 5.756278493s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X............X.X....X........X...........................] duration: 3.590765096s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:36:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590790603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X............X.X....X........X..........................] duration: 3.368934757s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:36:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.368964082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.X....X.X....X.X..XX.XXX...X.XXXXX..X..X...X....X.X......] duration: 5.671152046s trace: https://jaeger.dev.storj.io/trace/62601e16b7e3acab FAIL (2025-12-30 05:35:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:31:25.686772372 (590h2m25.557855417s) [upload=517249]: Starting upload Dec 30 05:31:25.686864357 (590h2m25.557947402s) [upload=517249]: Done starting upload Dec 30 05:31:25.686967083 (590h2m25.558050128s) [upload=517249]: Uploading segments... Dec 30 05:31:25.687144340 (590h2m25.558227395s) [upload=517249]: Got next segment. Inline: false Dec 30 05:31:25.687194385 (590h2m25.55827743s) [upload=517249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:31:25.687199926 (590h2m25.558282972s) [upload=517249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:31:25.934194068 (590h2m25.805277123s) [seg_pos={0 0}, upload=517249]: Begin upload segment... Dec 30 05:31:25.934325408 (590h2m25.805408443s) [seg_pos={0 0}, upload=517249]: Done begin upload segment. Dec 30 05:31:25.934365684 (590h2m25.80544875s) [noise=true , address=77.90.31.11:4553 , piece_id=EZYGMFLFH4JYQRGYKBBIZW7YOHMXR2HHUMJPQVSECEK6I5CLDPMA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.934514447 (590h2m25.805597493s) [noise=true , address=145.79.227.46:4601 , piece_id=GYKUVKJU7PFON6GQNNEBURILJURMNV4AMWPD4BSZELBPKHN3WUBQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935171739 (590h2m25.806254784s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935430823 (590h2m25.806513868s) [noise=true , address=145.79.227.4:4185 , piece_id=CHL4TSLOFWIYSI4YA23PCCCZMQSW4XKVEXZMQCBLFR22KQATESZA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.935641374 (590h2m25.806724409s) [noise=true , address=145.79.227.2:4114 , piece_id=NA7CRCYCYTU4C4EHV6ED6LHUWQRTB2IPIVPIURK4M3GHXYUJ4RHA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936078195 (590h2m25.80716123s) [noise=true , address=145.79.227.10:4402 , piece_id=ZIY5G3WUK3G6MJQYJGHTA25ZVVAM76W6O5O6LH3ZHBNAMKWEJXKQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936084498 (590h2m25.807167543s) [noise=true , address=145.79.227.3:4150 , piece_id=3YKCLWHRPVWF52O7L3Q3ID5Z7ZTIPNRLYYGWKQYSGF5BJB6SOWCQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936661626 (590h2m25.807744671s) [noise=true , address=45.91.116.14:4191 , piece_id=FOCBVJUUTKCRM74DS5RHRUS47W2Z2Y5A4OQSOGTJ6EC7OB4V6YGA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.936682927 (590h2m25.807765972s) [noise=true , address=145.79.227.26:4653 , piece_id=NA5MU225IFHVDEDZEEUFOBOWFBK7EB25XP3WNOWOZVEL3LOGFYRA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937168342 (590h2m25.808251387s) [noise=true , address=145.79.227.28:4725 , piece_id=ZWA5DQN6YAG6VYMLVIONKNCIAWZ6GT7URVPM2QVAO4INRSZLUKAQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937452473 (590h2m25.808535538s) [noise=true , address=145.79.227.18:4006 , piece_id=DHUB7DOY225PR7T5BFMCDDE5JD5FS7O5TZMDS5IWNR34JMD3JIEA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.937758335 (590h2m25.80884138s) [noise=true , address=145.79.227.36:4517 , piece_id=VHMRDJJXAPLM6X5LEXYMAB6D2EWAI7EVITX4RDPM6HQ5HEW5IVMQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.938376161 (590h2m25.809459206s) [noise=true , address=45.91.116.6:4973 , piece_id=J4E3DDN7XFTQ3T3OQ3VOP7LVCGOFH4QKXIUXCRIURR7XIR5KVUXQ, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.938383937 (590h2m25.809466972s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.939067167 (590h2m25.810150212s) [noise=true , address=145.79.227.50:4374 , piece_id=RXGBIGU5X3VPDQKTJCBGHG62LZ5RWLECB3QIT5FUL73U3FO5R77Q, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:25.939756139 (590h2m25.810839194s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Uploading piece... Dec 30 05:31:27.639130403 (590h2m27.510213438s) [upload=517249]: Next returned nil segment Dec 30 05:31:27.639148108 (590h2m27.510231153s) [upload=517249]: Waiting for error group managing segments... Dec 30 05:31:27.918866023 (590h2m27.789949078s) [noise=true , address=145.79.227.36:4517 , piece_id=VHMRDJJXAPLM6X5LEXYMAB6D2EWAI7EVITX4RDPM6HQ5HEW5IVMQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.919204248 (590h2m27.790287283s) [noise=true , address=145.79.227.4:4185 , piece_id=CHL4TSLOFWIYSI4YA23PCCCZMQSW4XKVEXZMQCBLFR22KQATESZA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.923402212 (590h2m27.794485257s) [noise=true , address=145.79.227.26:4653 , piece_id=NA5MU225IFHVDEDZEEUFOBOWFBK7EB25XP3WNOWOZVEL3LOGFYRA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.930735455 (590h2m27.8018185s) [noise=true , address=145.79.227.28:4725 , piece_id=ZWA5DQN6YAG6VYMLVIONKNCIAWZ6GT7URVPM2QVAO4INRSZLUKAQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.938066956 (590h2m27.809149991s) [noise=true , address=145.79.227.18:4006 , piece_id=DHUB7DOY225PR7T5BFMCDDE5JD5FS7O5TZMDS5IWNR34JMD3JIEA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.942031345 (590h2m27.8131144s) [noise=true , address=145.79.227.50:4374 , piece_id=RXGBIGU5X3VPDQKTJCBGHG62LZ5RWLECB3QIT5FUL73U3FO5R77Q, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.942575931 (590h2m27.813658966s) [noise=true , address=145.79.227.3:4150 , piece_id=3YKCLWHRPVWF52O7L3Q3ID5Z7ZTIPNRLYYGWKQYSGF5BJB6SOWCQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.943193638 (590h2m27.814276683s) [noise=true , address=145.79.227.46:4601 , piece_id=GYKUVKJU7PFON6GQNNEBURILJURMNV4AMWPD4BSZELBPKHN3WUBQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.963317117 (590h2m27.834400152s) [noise=true , address=145.79.227.2:4114 , piece_id=NA7CRCYCYTU4C4EHV6ED6LHUWQRTB2IPIVPIURK4M3GHXYUJ4RHA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:27.995157196 (590h2m27.866240231s) [noise=true , address=145.79.227.10:4402 , piece_id=ZIY5G3WUK3G6MJQYJGHTA25ZVVAM76W6O5O6LH3ZHBNAMKWEJXKQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.227384079 (590h2m28.098467114s) [noise=true , address=77.90.31.11:4553 , piece_id=EZYGMFLFH4JYQRGYKBBIZW7YOHMXR2HHUMJPQVSECEK6I5CLDPMA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.237871599 (590h2m28.108954634s) [noise=true , address=45.91.116.6:4973 , piece_id=J4E3DDN7XFTQ3T3OQ3VOP7LVCGOFH4QKXIUXCRIURR7XIR5KVUXQ, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.380368961 (590h2m28.251451996s) [noise=true , address=45.91.116.14:4191 , piece_id=FOCBVJUUTKCRM74DS5RHRUS47W2Z2Y5A4OQSOGTJ6EC7OB4V6YGA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: <nil> Dec 30 05:31:28.380391995 (590h2m28.25147504s) [seg_pos={0 0}, upload=517249]: Segment reached optimal threshold of 13 pieces. Dec 30 05:31:28.380585093 (590h2m28.251668138s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 05:31:28.380594079 (590h2m28.251677134s) [noise=true , address=145.79.227.49:4337 , piece_id=537FLBIPFGTJT3X4BWVONPUGA347DACZOWAWXQ45RC7M7IC2SLKA, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380645388 (590h2m28.251728423s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49420->45.91.116.8:4047: use of closed network connection Dec 30 05:31:28.380656158 (590h2m28.251739223s) [noise=true , address=45.91.116.8:4047 , piece_id=XC3BLLLIOVCJIWRQ2QQ3LICAFMCYDLN2VI5E7AXJKISCEROKW4NA, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380848304 (590h2m28.251931339s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 05:31:28.380858403 (590h2m28.251941438s) [noise=true , address=77.90.31.13:4627 , piece_id=KT76MANWGHXOQPFGNBXIQS5T52FOIXF7U5ML7YHNSIYC4U3MIM2Q, seg_pos={0 0}, upload=517249]: Long tail cancellation due to optimal threshold Dec 30 05:31:28.380879233 (590h2m28.251962278s) [seg_pos={0 0}, upload=517249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:31:28.380965317 (590h2m28.252048362s) [upload=517249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:31:28.380986517 (590h2m28.252069562s) [upload=517249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:31:28.668586000 (590h2m28.539669035s) [upload=517249]: Done uploading segments... Dec 30 05:31:28.668609435 (590h2m28.53969247s) [upload=517249]: Upload finished. err: <nil> Duration: 5.671177333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 15.902477758s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:35:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 15.902584391s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)