Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX..X.............X.................X.......X............] duration: 11.12503494s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 11.125080757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.678538837s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.678575245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX..X.............X.................X.......X...........] duration: 5.945388566s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:09:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 5.945407643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.450467703s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:09:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450493543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.543143502s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:09:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543173762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX..X.............X.................X.......X..........] duration: 6.571856916s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:08:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.571882624s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.517796289s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:08:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517824832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX..X.............X.................X.......X.........] duration: 6.780835922s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:07:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.780859426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX..X.............X.................X.......X........] duration: 6.724215706s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:07:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.724244701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.773482366s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:07:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.773512113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX..X.............X.................X.......X.......] duration: 6.390338385s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:07:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.390359644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX..X.............X.................X.......X......] duration: 5.678143566s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:06:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 5.678172912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.484537436s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485060081s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.581157076s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:05:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581176943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX..X.............X.................X.......X.....] duration: 7.357442942s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 7.357472639s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.510027715s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:04:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510055048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX..X.............X.................X.......X....] duration: 6.747540666s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:04:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.747567548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.523975521s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524032748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XX..X.............X.................X.......X...] duration: 6.977276734s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.977305147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.55431996s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:03:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554343126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X............XX..X.............X.................X.......X..] duration: 14.225975522s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:02:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 14.226003928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.581985972s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582014346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X............XX..X.............X.................X.......X.] duration: 6.474282558s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 6.474312784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X............XX..X.............X.................X.......X] duration: 30.003357624s trace: https://jaeger.dev.storj.io/trace/648ca1441779a629 FAIL (2025-12-23 15:00:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A) context deadline exceeded ======LOGS====== Dec 23 15:00:26.804499367 (431h31m26.675582412s) [upload=378811]: Starting upload Dec 23 15:00:26.804599949 (431h31m26.675682994s) [upload=378811]: Done starting upload Dec 23 15:00:26.804687816 (431h31m26.675770861s) [upload=378811]: Uploading segments... Dec 23 15:00:26.804890492 (431h31m26.675973537s) [upload=378811]: Got next segment. Inline: false Dec 23 15:00:26.804931530 (431h31m26.676014565s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:00:26.804937301 (431h31m26.676020357s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:00:27.048121346 (431h31m26.919204391s) [seg_pos={0 0}, upload=378811]: Begin upload segment... Dec 23 15:00:27.048178013 (431h31m26.919261058s) [seg_pos={0 0}, upload=378811]: Done begin upload segment. Dec 23 15:00:27.048210966 (431h31m26.919294011s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049330397 (431h31m26.920413443s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049384862 (431h31m26.920467907s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049897137 (431h31m26.920980182s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.049906164 (431h31m26.920989209s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050294514 (431h31m26.921377549s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050348056 (431h31m26.921431101s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.050655771 (431h31m26.921738816s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051006840 (431h31m26.922089895s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051134232 (431h31m26.922217278s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051504077 (431h31m26.922587122s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.051569502 (431h31m26.922652547s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052274163 (431h31m26.923357208s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.052806477 (431h31m26.923889522s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.053343981 (431h31m26.924427026s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:27.054144094 (431h31m26.925227129s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Uploading piece... Dec 23 15:00:28.941475985 (431h31m28.81255902s) [upload=378811]: Next returned nil segment Dec 23 15:00:28.941489692 (431h31m28.812572737s) [upload=378811]: Waiting for error group managing segments... Dec 23 15:00:29.317169139 (431h31m29.188252184s) [noise=true , address=45.91.116.13:4153 , piece_id=GJCGHWH3DSJRTXSHMQKIJFXTSZCH4OVX7QL63NS5GZJJ47DBWBEQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.339540287 (431h31m29.210623332s) [noise=true , address=77.90.31.17:4773 , piece_id=ATJHSQG62VMJSTVKOFOK33PXE6X2YTO357CKJVQ6QZY7N5KVUADQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.369471081 (431h31m29.240554126s) [noise=true , address=45.91.116.4:4937 , piece_id=YYAEJB3EYJG3RTUKOGDMWQEGCLRMDUBZYVHSBHTQDRO4X7OV3HNA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.423280974 (431h31m29.294364009s) [noise=true , address=45.91.116.7:4010 , piece_id=SFX7NF35H3ATSRUKERLWZUB5LXWO7M4U6IQG7J2EKBFYESEG4XBQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.428068791 (431h31m29.299151836s) [noise=true , address=77.90.31.16:4736 , piece_id=JI26QMRBKWJG2IU2MGWIA6HRX4SJWWEUVLTTAZVY4ABVPS5HP45A, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.703600030 (431h31m29.574683075s) [noise=true , address=145.79.227.32:4835 , piece_id=PHWC6M5C756BC4FHKOQ7QE2KWWP2BG66IJX2J7RWAPNHXPGQFQCA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.712306157 (431h31m29.583389202s) [noise=true , address=145.79.227.39:4013 , piece_id=GMIN7WZT3SXX3UJ22ND2Q56UVC54HZERY7NMPD3EF56Q4BFTJR6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713087214 (431h31m29.584170259s) [noise=true , address=145.79.227.20:4438 , piece_id=7RGOQYVWQ2GXHMVSKYPNVR5TQRBC4UVE4FV6DGCN3BSHHRNGXSKA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.713861079 (431h31m29.584944114s) [noise=true , address=145.79.227.21:4473 , piece_id=MNUKYQZCKMZFMYNFLRE6LZVASYGU4AR4A4OMPE6Y7XA3WYGD7ECQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.743742678 (431h31m29.614825723s) [noise=true , address=145.79.227.45:4230 , piece_id=KYXAATIPB7IGQ6W6NRT3JPGRNN3ZPXRCPWGFE76R2VT4YX65CTUQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.744478268 (431h31m29.615561314s) [noise=true , address=145.79.227.35:4481 , piece_id=DFI42SM6MH5BYUUTLRJMJILVSXA5Q5TXGRXVOSXUAM4HIUT7ZS6Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.935543569 (431h31m29.806626614s) [noise=true , address=145.79.227.26:4655 , piece_id=VMLULDXH2DMAX4NBTQLZIDQB5WLFXUPAQZ77AJGGQT6EGY6ZY65Q, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953600503 (431h31m29.824683538s) [noise=true , address=145.79.227.50:4375 , piece_id=CYIWDJNFOJMWUM3ZXCQSBZOCIBVIPPBULTIESFQCIDN4JY3P5PYQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: <nil> Dec 23 15:00:29.953666459 (431h31m29.824749504s) [seg_pos={0 0}, upload=378811]: Segment reached optimal threshold of 13 pieces. Dec 23 15:00:29.953878532 (431h31m29.824961577s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.953885816 (431h31m29.824968851s) [noise=true , address=145.79.227.38:4978 , piece_id=GQEIXCJTSIWB5NWZOE2ZMJR22HGOHZZGSXUBXX722IO5KJSZ67JA, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954189054 (431h31m29.825272099s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954196618 (431h31m29.825279653s) [noise=true , address=145.79.227.2:4113 , piece_id=F4TZ6F5S4DPZERLBTO3DLCYTSX42NR4N3LYBKGK735SHME2DGKJQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954315985 (431h31m29.82539902s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 23 15:00:29.954321596 (431h31m29.825404631s) [noise=true , address=145.79.227.46:4603 , piece_id=XHAP2WAAIZJH6LH3VLRFWVSANLW3NP5NJ6BOYJDFUDSAJAVDUUDQ, seg_pos={0 0}, upload=378811]: Long tail cancellation due to optimal threshold Dec 23 15:00:29.954357785 (431h31m29.82544083s) [seg_pos={0 0}, upload=378811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:00:29.954428890 (431h31m29.825511926s) [upload=378811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:00:29.954433559 (431h31m29.825516594s) [upload=378811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:00:30.249569558 (431h31m30.120652593s) [upload=378811]: Done uploading segments... Dec 23 15:00:30.249589226 (431h31m30.120672271s) [upload=378811]: Upload finished. err: <nil> Duration: 30.003405807s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X....................................................X......] duration: 3.413605747s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:00:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413635465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X............XX..X.............X.................X.......] duration: 6.245490938s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:59:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 6.245515576s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X....................................................X.....] duration: 3.57028969s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:59:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570323104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX..X............XX..X.............X.................X......] duration: 6.722002657s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:59:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 6.722028125s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X....................................................X....] duration: 3.724562391s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:58:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.724584021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX..X............XX..X.............X.................X.....] duration: 11.533005795s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:58:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 11.533027257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X....................................................X...] duration: 3.576401626s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:58:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576425632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.XX..X............XX..X.............X.................X....] duration: 6.332961819s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 6.332990213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.XX..X............XX..X.............X.................X...] duration: 6.357703531s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:57:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 6.357731614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X....................................................X..] duration: 3.727576843s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:57:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.727605105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X....................................................X.] duration: 3.486766189s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:56:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486789723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.XX..X............XX..X.............X.................X..] duration: 6.076974986s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:56:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 6.077001296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X....................................................X] duration: 5.001404866s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 14:55:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001426117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.XX..X............XX..X.............X.................X.] duration: 7.75446372s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:55:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 7.754487855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.XX..X............XX..X.............X.................X] duration: 30.003087341s trace: https://jaeger.dev.storj.io/trace/1e3ed3ef1d4f2394 FAIL (2025-12-23 14:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA) context deadline exceeded ======LOGS====== Dec 23 14:54:45.301019547 (431h25m45.172102582s) [upload=378724]: Starting upload Dec 23 14:54:45.301072577 (431h25m45.172155612s) [upload=378724]: Done starting upload Dec 23 14:54:45.301181775 (431h25m45.17226481s) [upload=378724]: Uploading segments... Dec 23 14:54:45.301229266 (431h25m45.172312311s) [upload=378724]: Got next segment. Inline: false Dec 23 14:54:45.301258551 (431h25m45.172341596s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:54:45.301262750 (431h25m45.172345785s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:54:45.545446587 (431h25m45.416529632s) [seg_pos={0 0}, upload=378724]: Begin upload segment... Dec 23 14:54:45.545496122 (431h25m45.416579167s) [seg_pos={0 0}, upload=378724]: Done begin upload segment. Dec 23 14:54:45.545512262 (431h25m45.416595298s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.545526370 (431h25m45.416609415s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546002235 (431h25m45.41708528s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546372561 (431h25m45.417455606s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.546742586 (431h25m45.417825631s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547173145 (431h25m45.41825619s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547465391 (431h25m45.418548427s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.547786002 (431h25m45.418869047s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548112574 (431h25m45.419195609s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548403127 (431h25m45.419486162s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.548708158 (431h25m45.419791203s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.549047845 (431h25m45.42013089s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.550270644 (431h25m45.421353679s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.551577872 (431h25m45.422660907s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.552776966 (431h25m45.423860001s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:45.553952174 (431h25m45.425035209s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Uploading piece... Dec 23 14:54:47.304578194 (431h25m47.175661229s) [upload=378724]: Next returned nil segment Dec 23 14:54:47.304591670 (431h25m47.175674705s) [upload=378724]: Waiting for error group managing segments... Dec 23 14:54:47.586702467 (431h25m47.457785512s) [noise=true , address=145.79.227.49:4338 , piece_id=RELVLJHASGFIVW3GT6CXJ66BNYH64CJOUA5H6SBSEQK4NGHNRTLQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598305491 (431h25m47.469388526s) [noise=true , address=145.79.227.2:4113 , piece_id=6PIXFIGNFE4FENWQGMN55VYG56C3L5YJS4NHBJZXNEGMZKKR7ECA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.598443484 (431h25m47.469526519s) [noise=true , address=145.79.227.41:4087 , piece_id=6YISCJTQC7WLTX55I5X4FFGPJ42ZZEVWWFASTRMLISNED77G6ILA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.603123997 (431h25m47.474207042s) [noise=true , address=145.79.227.35:4483 , piece_id=WB6VPVTO32XA2XFXMY6CMCHM6RVVHVQKSMLMLOKJ5WCSHXB5FO6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.616478776 (431h25m47.487561831s) [noise=true , address=145.79.227.48:4302 , piece_id=77APZQJRGEG4C4JUVLIIKQCH6QYUSWHVDCOAA5ZMHV52ITGYAB5Q, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.814900525 (431h25m47.68598356s) [noise=true , address=77.90.31.17:4773 , piece_id=SWIRLYDWXKQBCPSBDNJA2WQD7AKL3RQUYFPG7U2ZSHE45TWEQRJA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.822858087 (431h25m47.693941122s) [noise=true , address=45.91.116.11:4117 , piece_id=TAKZMJ34EB22THPY5FBUUTEGMCXN3467U4NLDCZIIDI6M6UGRYXA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840095871 (431h25m47.711178916s) [noise=true , address=145.79.227.51:4411 , piece_id=QUSTLAVTOYHOSGNP7YJVGXFX2GWBRLPMOK4FWV4OUNHCSMIAHCAQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.840480553 (431h25m47.711563588s) [noise=true , address=145.79.227.46:4601 , piece_id=EZXA273XXAYAWUKZZY3UJMO2JP53VLHGNMPUNGJ6ZWCMZPNYKTFA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:47.896871151 (431h25m47.767954186s) [noise=true , address=45.91.116.13:4156 , piece_id=MY7XPMIFIUREV4ZUD4KSECF5AXQJXNQOAUCFEPGQS6NCSSGS6Y6A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.040148155 (431h25m47.91123119s) [noise=true , address=145.79.227.43:4157 , piece_id=GZ7YIHTDSS3EC7FZJEPZ7UPZKEKB4PQUTSI3U2PJTN2NKMXHSU3A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.052350630 (431h25m47.923433665s) [noise=true , address=145.79.227.23:4545 , piece_id=3XFXN5GZZ353NQLADZ4ZDHBBUNFLOU4WZJRD2HINM545SCWEC44A, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053167115 (431h25m47.92425015s) [noise=true , address=145.79.227.7:4293 , piece_id=5XIYU43QCAFKCEUWSFWRKE5LE26GVRQW7KGSIRCJBXFVNALSR5QA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: <nil> Dec 23 14:54:48.053175651 (431h25m47.924258686s) [seg_pos={0 0}, upload=378724]: Segment reached optimal threshold of 13 pieces. Dec 23 14:54:48.053337790 (431h25m47.924420826s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053343601 (431h25m47.924426656s) [noise=true , address=45.91.116.5:4297 , piece_id=CUASCCCJPC4OCJW7SPUJ5PWD3GNO2UV6TOF2TYOROBXOQZQC5QZQ, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053447840 (431h25m47.924530875s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053452889 (431h25m47.924535934s) [noise=true , address=145.79.227.32:4836 , piece_id=J7FX3ESNOG62L572LI7PWELHKHELMDO37MV6XDYD7RDEP5OVQRGA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053616371 (431h25m47.924699406s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 14:54:48.053637891 (431h25m47.924720926s) [noise=true , address=77.90.31.16:4735 , piece_id=5M2UZBTE32TJ3KEIOJF6NHTGOW46YCNKBLUOHTZ36EJWBDQDYGMA, seg_pos={0 0}, upload=378724]: Long tail cancellation due to optimal threshold Dec 23 14:54:48.053655335 (431h25m47.92473838s) [seg_pos={0 0}, upload=378724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:54:48.053736810 (431h25m47.924819845s) [upload=378724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:54:48.053741609 (431h25m47.924824644s) [upload=378724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:54:48.347554339 (431h25m48.218637374s) [upload=378724]: Done uploading segments... Dec 23 14:54:48.347574687 (431h25m48.218657752s) [upload=378724]: Upload finished. err: <nil> Duration: 30.003114582s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.560530114s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:54:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560552177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.576109571s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:54:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576128026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.XX..X............XX..X.............X.................] duration: 7.914206895s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:53:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 7.914240339s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.669611115s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:52:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.669666039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.XX..X............XX..X.............X................] duration: 6.226795318s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.226823001s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.560891683s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:51:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560917884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.XX..X............XX..X.............X...............] duration: 6.239279159s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 6.239306642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 4.2582635s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:51:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.258290051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.XX..X............XX..X.............X..............] duration: 5.926900057s trace: https://jaeger.dev.storj.io/trace/74788274d19d7a46 FAIL (2025-12-23 14:50:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA) context deadline exceeded ======LOGS====== Dec 23 14:40:11.868119876 (431h11m11.739202911s) [upload=378516]: Starting upload Dec 23 14:40:11.868207653 (431h11m11.739290718s) [upload=378516]: Done starting upload Dec 23 14:40:11.868374521 (431h11m11.739457566s) [upload=378516]: Uploading segments... Dec 23 14:40:11.868506252 (431h11m11.739589297s) [upload=378516]: Got next segment. Inline: false Dec 23 14:40:11.868563651 (431h11m11.739646696s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 14:40:11.868569853 (431h11m11.739652898s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 14:40:12.111082779 (431h11m11.982165825s) [seg_pos={0 0}, upload=378516]: Begin upload segment... Dec 23 14:40:12.111134018 (431h11m11.982217053s) [seg_pos={0 0}, upload=378516]: Done begin upload segment. Dec 23 14:40:12.111151831 (431h11m11.982234866s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111205434 (431h11m11.982288479s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111360929 (431h11m11.982443974s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.111802781 (431h11m11.982885826s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112248239 (431h11m11.983331294s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.112648541 (431h11m11.983731586s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113366068 (431h11m11.984449113s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.113765128 (431h11m11.984848163s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114301239 (431h11m11.985384304s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.114637028 (431h11m11.985720063s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115428274 (431h11m11.986511319s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.115717145 (431h11m11.98680018s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116286669 (431h11m11.987369714s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.116830806 (431h11m11.987913841s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117007933 (431h11m11.988090968s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:12.117627111 (431h11m11.988710146s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Uploading piece... Dec 23 14:40:13.362711746 (431h11m13.233794791s) [noise=true , address=77.90.31.15:4697 , piece_id=LRYC2NIG3MUVOMATXPEG4QMMIWKVED3HCBOQ6C7CXAJCSWVXQVTA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 23 14:40:13.824993954 (431h11m13.696076989s) [upload=378516]: Next returned nil segment Dec 23 14:40:13.825024922 (431h11m13.696107957s) [upload=378516]: Waiting for error group managing segments... Dec 23 14:40:14.102416555 (431h11m13.9734996s) [noise=true , address=145.79.227.41:4085 , piece_id=PJ5H7E2LNKRAEAOFEWIVEV4USLM7DJ4XYD4GSQT5N5SQPWKLRLXA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.104229709 (431h11m13.975312744s) [noise=true , address=145.79.227.4:4187 , piece_id=LIXR5F3EXJTP6ALXC3A3V54IKIAK634JCBRONOCFN6RYI5KGOBHQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108343191 (431h11m13.979426236s) [noise=true , address=145.79.227.50:4373 , piece_id=63RBNMEQ2IDHDV73J4FXZHB37KECBX6SWVNRLM36V5EPWKYTKYMQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.108974293 (431h11m13.980057338s) [noise=true , address=145.79.227.11:4730 , piece_id=M5ZYH4B3UNIY65JJFGGRVXBR2VIOEVEJSK4AFO6SMBEFX2ZXYXYQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.115937562 (431h11m13.987020607s) [noise=true , address=145.79.227.37:4565 , piece_id=K62BRPWTKPSD4IJD2NS2P7QC4JZ3QNLH42GO7XZKI2EAXF62K4KQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.124740384 (431h11m13.995823419s) [noise=true , address=145.79.227.39:4013 , piece_id=ENCWF72DYPR54VZN4KXDT3U2CS54BJM2BQQ7BYA5ZP373FDWXZVA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.127790853 (431h11m13.998873898s) [noise=true , address=145.79.227.42:4123 , piece_id=P53WTGLAZA2SBX7MEIMYJABAI76T552NUANBWBNXCDAUNR426CSQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.185052439 (431h11m14.056135484s) [noise=true , address=145.79.227.49:4339 , piece_id=PQFUWEUSC7MOP6NNXIOE5M4DFA5OIVRLI3NW2DA6DATVCWQMUGJA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190499301 (431h11m14.061582346s) [noise=true , address=145.79.227.45:4231 , piece_id=ZLYVEFQLJXGZ4PJ6HRXJZLPQWDWTOH7YAZTH3CCMVXUANT2DVGMA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.190648225 (431h11m14.06173127s) [noise=true , address=145.79.227.14:4839 , piece_id=KKNTSJISRHZKMTMDT75F2QDQF4EV6PTQ46QQTOLWFOGX5E5LDWOQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.382679737 (431h11m14.253762772s) [noise=true , address=45.91.116.9:4261 , piece_id=PZIQACPNQHHQD4MILGT6KPOOKU4D6O5FPG4D4TWQOVYLCGOPPAGQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.487075846 (431h11m14.358158881s) [noise=true , address=77.90.31.13:4627 , piece_id=2QPC7ELY3YBFYD7VUV7RHIPFY5PICWG5ZQMLSZRZKSD43Y6LBBOA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500579970 (431h11m14.371663005s) [noise=true , address=45.91.116.12:4228 , piece_id=PVTAKKVVQ427X5LMVEEJQVOO22NKBC6YRALCX3FYXOFJIBZVS3HQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: <nil> Dec 23 14:40:14.500606741 (431h11m14.371689786s) [seg_pos={0 0}, upload=378516]: Segment reached optimal threshold of 13 pieces. Dec 23 14:40:14.500868379 (431h11m14.371951424s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60010->45.91.116.5:4298: use of closed network connection Dec 23 14:40:14.500879801 (431h11m14.371962867s) [noise=true , address=45.91.116.5:4298 , piece_id=LNIEPYC6VOANJJAHWGIR54C2NSQNVBLW56WLJS3ATE3RMQI2WTQQ, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501132863 (431h11m14.372215898s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 23 14:40:14.501169122 (431h11m14.372252167s) [noise=true , address=45.91.116.13:4155 , piece_id=BJIO3BI6TOOJLAO7UWOWDEWS2YA7PHTD6XAQZBJ4R344R5FHXIQA, seg_pos={0 0}, upload=378516]: Long tail cancellation due to optimal threshold Dec 23 14:40:14.501199750 (431h11m14.372282786s) [seg_pos={0 0}, upload=378516]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 14:40:14.501347652 (431h11m14.372430707s) [upload=378516]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 14:40:14.501356418 (431h11m14.372439463s) [upload=378516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 14:40:14.797729085 (431h11m14.66881212s) [upload=378516]: Done uploading segments... Dec 23 14:40:14.797748543 (431h11m14.668831588s) [upload=378516]: Upload finished. err: <nil> Duration: 5.926926719s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.669100102s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:50:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.669129938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X............X..............................................] duration: 3.610231647s trace: https://jaeger.dev.storj.io/trace/772f273c480c0b2b FAIL (2025-12-23 14:49:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.610323403s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)