Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.XXX.X.....X.......X...X.................................X.] duration: 5.713314986s trace: https://jaeger.dev.storj.io/trace/1da21c3c29ae4c27 FAIL (2025-12-26 07:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA) context deadline exceeded ======LOGS====== Dec 26 07:16:45.967091946 (495h47m45.838174991s) [upload=435168]: Starting upload Dec 26 07:16:45.967170195 (495h47m45.83825323s) [upload=435168]: Done starting upload Dec 26 07:16:45.967233996 (495h47m45.838317031s) [upload=435168]: Uploading segments... Dec 26 07:16:45.967349997 (495h47m45.838433032s) [upload=435168]: Got next segment. Inline: false Dec 26 07:16:45.967380635 (495h47m45.83846367s) [upload=435168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:16:45.967386887 (495h47m45.838469932s) [upload=435168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:16:46.211695672 (495h47m46.082778707s) [seg_pos={0 0}, upload=435168]: Begin upload segment... Dec 26 07:16:46.211737782 (495h47m46.082820817s) [seg_pos={0 0}, upload=435168]: Done begin upload segment. Dec 26 07:16:46.211762579 (495h47m46.082845614s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.211889010 (495h47m46.082972046s) [noise=true , address=145.79.227.18:4007 , piece_id=R6NE6CSUJKTRCUAXZV2EKDJSFWXOLVA3LGMNC5UKDGHT6IWRRG2Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212327445 (495h47m46.08341048s) [noise=true , address=145.79.227.43:4159 , piece_id=C4655HKU7YDKPUIJGSPJOJKOYW43MFKY5I2FPWAC44NXLW7LJR2Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212632466 (495h47m46.083715531s) [noise=true , address=145.79.227.9:4366 , piece_id=AZUPRTUUF2IQLRDDF5Q2PAA2QMUBJHUQ7G4H32BPKDU3SCE4UVZQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212949830 (495h47m46.084032865s) [noise=true , address=145.79.227.17:4970 , piece_id=A4B5CSH676LCCMF2VEQWHHTR7VO7KHHDEGDCQ75LQSD7A2YWLVYA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213214945 (495h47m46.08429799s) [noise=true , address=145.79.227.52:4445 , piece_id=N62AGLGYFB6J7DL4EZO2YFHU3NLGW6Z33DIQWCRDY35NPKO2A64Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213479198 (495h47m46.084562233s) [noise=true , address=145.79.227.14:4838 , piece_id=XTK3CRKGN3RMB3H2CYDPPGPYBI5O5XOHPGS274N7G4GUX7L5QRHA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213741107 (495h47m46.084824142s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.214279001 (495h47m46.085362046s) [noise=true , address=145.79.227.4:4185 , piece_id=MTVVQ56RTGC6DPPJS6IABIRPWUKX332M2US2QDDECM6F73VZLY7Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.215706719 (495h47m46.086789764s) [noise=true , address=77.90.31.17:4773 , piece_id=NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.216825259 (495h47m46.087908294s) [noise=true , address=77.90.31.10:4518 , piece_id=BL6XL3JHHQQ3QRMNO6FP4VJ5TCU7JVI5I5PHV3W2Z27TDR3NIWPA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.217968576 (495h47m46.089051611s) [noise=true , address=77.90.31.19:4844 , piece_id=WDLFH4RDI7L32SOABLXI5H7S366DUJPYHSRXLZWFTV2UE24JJK2A, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.219572701 (495h47m46.090655736s) [noise=true , address=77.90.31.14:4664 , piece_id=3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.221069962 (495h47m46.092152997s) [noise=true , address=145.79.227.36:4518 , piece_id=O2LAKZHBY3NKLMNRJ4HY5FQRAH5PWRPY6XEEFUQYVUZUMKT5FOVA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.222246623 (495h47m46.093329658s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.223169070 (495h47m46.094252115s) [noise=true , address=45.91.116.7:4012 , piece_id=CRVRYCV2UMKVPEFZULQNMCWAWJWSEVBRSNPXD5ON2NQADIBTM44A, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:47.932347861 (495h47m47.803430896s) [upload=435168]: Next returned nil segment Dec 26 07:16:47.932361227 (495h47m47.803444262s) [upload=435168]: Waiting for error group managing segments... Dec 26 07:16:48.217956288 (495h47m48.089039323s) [noise=true , address=145.79.227.9:4366 , piece_id=AZUPRTUUF2IQLRDDF5Q2PAA2QMUBJHUQ7G4H32BPKDU3SCE4UVZQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.218440569 (495h47m48.089523614s) [noise=true , address=145.79.227.52:4445 , piece_id=N62AGLGYFB6J7DL4EZO2YFHU3NLGW6Z33DIQWCRDY35NPKO2A64Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.219483505 (495h47m48.09056654s) [noise=true , address=145.79.227.14:4838 , piece_id=XTK3CRKGN3RMB3H2CYDPPGPYBI5O5XOHPGS274N7G4GUX7L5QRHA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.221855002 (495h47m48.092938037s) [noise=true , address=145.79.227.4:4185 , piece_id=MTVVQ56RTGC6DPPJS6IABIRPWUKX332M2US2QDDECM6F73VZLY7Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.222357238 (495h47m48.093440273s) [noise=true , address=145.79.227.17:4970 , piece_id=A4B5CSH676LCCMF2VEQWHHTR7VO7KHHDEGDCQ75LQSD7A2YWLVYA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.223354688 (495h47m48.094437733s) [noise=true , address=145.79.227.36:4518 , piece_id=O2LAKZHBY3NKLMNRJ4HY5FQRAH5PWRPY6XEEFUQYVUZUMKT5FOVA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.230579995 (495h47m48.10166303s) [noise=true , address=145.79.227.18:4007 , piece_id=R6NE6CSUJKTRCUAXZV2EKDJSFWXOLVA3LGMNC5UKDGHT6IWRRG2Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.240211234 (495h47m48.111294269s) [noise=true , address=145.79.227.43:4159 , piece_id=C4655HKU7YDKPUIJGSPJOJKOYW43MFKY5I2FPWAC44NXLW7LJR2Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.405912207 (495h47m48.276995242s) [noise=true , address=77.90.31.14:4664 , piece_id=3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.408820374 (495h47m48.279903409s) [noise=true , address=77.90.31.19:4844 , piece_id=WDLFH4RDI7L32SOABLXI5H7S366DUJPYHSRXLZWFTV2UE24JJK2A, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.465554737 (495h47m48.336637772s) [noise=true , address=45.91.116.7:4012 , piece_id=CRVRYCV2UMKVPEFZULQNMCWAWJWSEVBRSNPXD5ON2NQADIBTM44A, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.487787422 (495h47m48.358870467s) [noise=true , address=77.90.31.17:4773 , piece_id=NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.523404381 (495h47m48.394487417s) [noise=true , address=77.90.31.10:4518 , piece_id=BL6XL3JHHQQ3QRMNO6FP4VJ5TCU7JVI5I5PHV3W2Z27TDR3NIWPA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.523429389 (495h47m48.394512434s) [seg_pos={0 0}, upload=435168]: Segment reached optimal threshold of 13 pieces. Dec 26 07:16:48.523655460 (495h47m48.394738505s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523663566 (495h47m48.394746601s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523775859 (495h47m48.394858904s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523786529 (495h47m48.394869564s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523923169 (495h47m48.395006204s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523932037 (495h47m48.395015072s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523951343 (495h47m48.395034378s) [seg_pos={0 0}, upload=435168]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:16:48.524025124 (495h47m48.395108159s) [upload=435168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:16:48.524032317 (495h47m48.395115362s) [upload=435168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:16:48.816379876 (495h47m48.687462911s) [upload=435168]: Done uploading segments... Dec 26 07:16:48.816402039 (495h47m48.687485084s) [upload=435168]: Upload finished. err: <nil> Duration: 5.713337139s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.491718936s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:17:50 UTC) orbiter warning: context deadline exceeded Duration: 3.491791454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.XXX.X.....X.......X...X.................................X] duration: 30.002878841s trace: https://jaeger.dev.storj.io/trace/1da21c3c29ae4c27 FAIL (2025-12-26 07:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA) context deadline exceeded ======LOGS====== Dec 26 07:16:45.967091946 (495h47m45.838174991s) [upload=435168]: Starting upload Dec 26 07:16:45.967170195 (495h47m45.83825323s) [upload=435168]: Done starting upload Dec 26 07:16:45.967233996 (495h47m45.838317031s) [upload=435168]: Uploading segments... Dec 26 07:16:45.967349997 (495h47m45.838433032s) [upload=435168]: Got next segment. Inline: false Dec 26 07:16:45.967380635 (495h47m45.83846367s) [upload=435168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:16:45.967386887 (495h47m45.838469932s) [upload=435168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:16:46.211695672 (495h47m46.082778707s) [seg_pos={0 0}, upload=435168]: Begin upload segment... Dec 26 07:16:46.211737782 (495h47m46.082820817s) [seg_pos={0 0}, upload=435168]: Done begin upload segment. Dec 26 07:16:46.211762579 (495h47m46.082845614s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.211889010 (495h47m46.082972046s) [noise=true , address=145.79.227.18:4007 , piece_id=R6NE6CSUJKTRCUAXZV2EKDJSFWXOLVA3LGMNC5UKDGHT6IWRRG2Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212327445 (495h47m46.08341048s) [noise=true , address=145.79.227.43:4159 , piece_id=C4655HKU7YDKPUIJGSPJOJKOYW43MFKY5I2FPWAC44NXLW7LJR2Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212632466 (495h47m46.083715531s) [noise=true , address=145.79.227.9:4366 , piece_id=AZUPRTUUF2IQLRDDF5Q2PAA2QMUBJHUQ7G4H32BPKDU3SCE4UVZQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.212949830 (495h47m46.084032865s) [noise=true , address=145.79.227.17:4970 , piece_id=A4B5CSH676LCCMF2VEQWHHTR7VO7KHHDEGDCQ75LQSD7A2YWLVYA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213214945 (495h47m46.08429799s) [noise=true , address=145.79.227.52:4445 , piece_id=N62AGLGYFB6J7DL4EZO2YFHU3NLGW6Z33DIQWCRDY35NPKO2A64Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213479198 (495h47m46.084562233s) [noise=true , address=145.79.227.14:4838 , piece_id=XTK3CRKGN3RMB3H2CYDPPGPYBI5O5XOHPGS274N7G4GUX7L5QRHA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.213741107 (495h47m46.084824142s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.214279001 (495h47m46.085362046s) [noise=true , address=145.79.227.4:4185 , piece_id=MTVVQ56RTGC6DPPJS6IABIRPWUKX332M2US2QDDECM6F73VZLY7Q, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.215706719 (495h47m46.086789764s) [noise=true , address=77.90.31.17:4773 , piece_id=NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.216825259 (495h47m46.087908294s) [noise=true , address=77.90.31.10:4518 , piece_id=BL6XL3JHHQQ3QRMNO6FP4VJ5TCU7JVI5I5PHV3W2Z27TDR3NIWPA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.217968576 (495h47m46.089051611s) [noise=true , address=77.90.31.19:4844 , piece_id=WDLFH4RDI7L32SOABLXI5H7S366DUJPYHSRXLZWFTV2UE24JJK2A, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.219572701 (495h47m46.090655736s) [noise=true , address=77.90.31.14:4664 , piece_id=3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.221069962 (495h47m46.092152997s) [noise=true , address=145.79.227.36:4518 , piece_id=O2LAKZHBY3NKLMNRJ4HY5FQRAH5PWRPY6XEEFUQYVUZUMKT5FOVA, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.222246623 (495h47m46.093329658s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:46.223169070 (495h47m46.094252115s) [noise=true , address=45.91.116.7:4012 , piece_id=CRVRYCV2UMKVPEFZULQNMCWAWJWSEVBRSNPXD5ON2NQADIBTM44A, seg_pos={0 0}, upload=435168]: Uploading piece... Dec 26 07:16:47.932347861 (495h47m47.803430896s) [upload=435168]: Next returned nil segment Dec 26 07:16:47.932361227 (495h47m47.803444262s) [upload=435168]: Waiting for error group managing segments... Dec 26 07:16:48.217956288 (495h47m48.089039323s) [noise=true , address=145.79.227.9:4366 , piece_id=AZUPRTUUF2IQLRDDF5Q2PAA2QMUBJHUQ7G4H32BPKDU3SCE4UVZQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.218440569 (495h47m48.089523614s) [noise=true , address=145.79.227.52:4445 , piece_id=N62AGLGYFB6J7DL4EZO2YFHU3NLGW6Z33DIQWCRDY35NPKO2A64Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.219483505 (495h47m48.09056654s) [noise=true , address=145.79.227.14:4838 , piece_id=XTK3CRKGN3RMB3H2CYDPPGPYBI5O5XOHPGS274N7G4GUX7L5QRHA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.221855002 (495h47m48.092938037s) [noise=true , address=145.79.227.4:4185 , piece_id=MTVVQ56RTGC6DPPJS6IABIRPWUKX332M2US2QDDECM6F73VZLY7Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.222357238 (495h47m48.093440273s) [noise=true , address=145.79.227.17:4970 , piece_id=A4B5CSH676LCCMF2VEQWHHTR7VO7KHHDEGDCQ75LQSD7A2YWLVYA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.223354688 (495h47m48.094437733s) [noise=true , address=145.79.227.36:4518 , piece_id=O2LAKZHBY3NKLMNRJ4HY5FQRAH5PWRPY6XEEFUQYVUZUMKT5FOVA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.230579995 (495h47m48.10166303s) [noise=true , address=145.79.227.18:4007 , piece_id=R6NE6CSUJKTRCUAXZV2EKDJSFWXOLVA3LGMNC5UKDGHT6IWRRG2Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.240211234 (495h47m48.111294269s) [noise=true , address=145.79.227.43:4159 , piece_id=C4655HKU7YDKPUIJGSPJOJKOYW43MFKY5I2FPWAC44NXLW7LJR2Q, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.405912207 (495h47m48.276995242s) [noise=true , address=77.90.31.14:4664 , piece_id=3QSGHMDEXROBKDWHCKLKHGCSXLOR3UNXXL4RDKVANRHKEFTC2DWQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.408820374 (495h47m48.279903409s) [noise=true , address=77.90.31.19:4844 , piece_id=WDLFH4RDI7L32SOABLXI5H7S366DUJPYHSRXLZWFTV2UE24JJK2A, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.465554737 (495h47m48.336637772s) [noise=true , address=45.91.116.7:4012 , piece_id=CRVRYCV2UMKVPEFZULQNMCWAWJWSEVBRSNPXD5ON2NQADIBTM44A, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.487787422 (495h47m48.358870467s) [noise=true , address=77.90.31.17:4773 , piece_id=NY6BOZEX7U537MQ6L4PNSI7OTWDETEAIOQEJEJPIAF3UUGNNDOMA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.523404381 (495h47m48.394487417s) [noise=true , address=77.90.31.10:4518 , piece_id=BL6XL3JHHQQ3QRMNO6FP4VJ5TCU7JVI5I5PHV3W2Z27TDR3NIWPA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: <nil> Dec 26 07:16:48.523429389 (495h47m48.394512434s) [seg_pos={0 0}, upload=435168]: Segment reached optimal threshold of 13 pieces. Dec 26 07:16:48.523655460 (495h47m48.394738505s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523663566 (495h47m48.394746601s) [noise=true , address=45.91.116.13:4154 , piece_id=T7XJAD2ZZDX7J376CQVVVUSP6AHU6RNX5ZFJDEP2O4RVEIZYQJWQ, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523775859 (495h47m48.394858904s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523786529 (495h47m48.394869564s) [noise=true , address=45.91.116.8:4046 , piece_id=42CTBSMNZVHF7ESUJMXF4B5NHIDKFQ7NADFSO5VBNOUMRRY2JGQA, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523923169 (495h47m48.395006204s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 07:16:48.523932037 (495h47m48.395015072s) [noise=true , address=77.90.31.11:4554 , piece_id=D5R5BPTPA5M5BCAUJB7UOMIG6VSNY6SCGNNNBBKZF7X27BPC3XIA, seg_pos={0 0}, upload=435168]: Long tail cancellation due to optimal threshold Dec 26 07:16:48.523951343 (495h47m48.395034378s) [seg_pos={0 0}, upload=435168]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:16:48.524025124 (495h47m48.395108159s) [upload=435168]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:16:48.524032317 (495h47m48.395115362s) [upload=435168]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:16:48.816379876 (495h47m48.687462911s) [upload=435168]: Done uploading segments... Dec 26 07:16:48.816402039 (495h47m48.687485084s) [upload=435168]: Upload finished. err: <nil> Duration: 30.002902917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.487217545s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:17:12 UTC) orbiter warning: context deadline exceeded Duration: 3.487244186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.523147493s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:16:03 UTC) orbiter warning: context deadline exceeded Duration: 3.523223568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X.XXX.X.....X.......X...X.................................] duration: 5.988188161s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:15:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.988212768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.575817865s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:15:25 UTC) orbiter warning: context deadline exceeded Duration: 3.575863192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X.XXX.X.....X.......X...X................................] duration: 7.849086634s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:15:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 7.849114456s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.464787466s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:15:03 UTC) orbiter warning: context deadline exceeded Duration: 3.46481665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.39177136s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:14:32 UTC) orbiter warning: context deadline exceeded Duration: 3.391808502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.X.XXX.X.....X.......X...X...............................] duration: 5.864317102s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.864338032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.724914058s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:13:56 UTC) orbiter warning: context deadline exceeded Duration: 3.725129709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.X.XXX.X.....X.......X...X..............................] duration: 5.761290431s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:13:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.761320337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 4.95821456s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:13:19 UTC) orbiter warning: context deadline exceeded Duration: 4.958239126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X.XXX.X.....X.......X...X.............................] duration: 6.155180452s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:13:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.155210638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.585994083s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:12:38 UTC) orbiter warning: context deadline exceeded Duration: 3.586013199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.X.XXX.X.....X.......X...X............................] duration: 6.621050448s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:11:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.621104912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.432816866s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:11:27 UTC) orbiter warning: context deadline exceeded Duration: 3.4328391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.X.XXX.X.....X.......X...X...........................] duration: 6.012624321s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:11:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.012652937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X.X.XXX.X.....X.......X...X..........................] duration: 6.06127506s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:10:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.061294337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.555254449s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:10:26 UTC) orbiter warning: context deadline exceeded Duration: 3.555291841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.......X.X.XXX.X.....X.......X...X.........................] duration: 6.034681695s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:10:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.034712232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X.X.XXX.X.....X.......X...X........................] duration: 6.021701298s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.021730302s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.471361584s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:09:46 UTC) orbiter warning: context deadline exceeded Duration: 3.471389397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.......X.X.XXX.X.....X.......X...X.......................] duration: 6.047908205s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.047935847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.......X.X.XXX.X.....X.......X...X......................] duration: 5.704193294s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:08:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.704221968s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.495060869s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:08:37 UTC) orbiter warning: context deadline exceeded Duration: 3.495089433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.......X.X.XXX.X.....X.......X...X.....................] duration: 15.147116681s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 15.147142389s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.605112275s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:08:03 UTC) orbiter warning: context deadline exceeded Duration: 3.605143245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.X.......X.X.XXX.X.....X.......X...X....................] duration: 6.073940814s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:07:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.073969298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.X.......X.X.XXX.X.....X.......X...X...................] duration: 5.757732912s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.757763561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.546197432s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:07:00 UTC) orbiter warning: context deadline exceeded Duration: 3.546212921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.X.......X.X.XXX.X.....X.......X...X..................] duration: 6.402838943s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:06:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.402866295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.51236572s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:05:59 UTC) orbiter warning: context deadline exceeded Duration: 3.512394115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X.X.......X.X.XXX.X.....X.......X...X.................] duration: 11.380252064s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:05:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 11.380284575s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.457560954s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:04:53 UTC) orbiter warning: context deadline exceeded Duration: 3.457624655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X.X.......X.X.XXX.X.....X.......X...X................] duration: 5.71807758s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.718127115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.43265594s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:04:10 UTC) orbiter warning: context deadline exceeded Duration: 3.432696748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.483216076s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:03:47 UTC) orbiter warning: context deadline exceeded Duration: 3.483292642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.......X.X.......X.X.XXX.X.....X.......X...X...............] duration: 5.98513909s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.985168436s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.510082494s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:03:28 UTC) orbiter warning: context deadline exceeded Duration: 3.510110566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......X.X.......X.X.XXX.X.....X.......X...X..............] duration: 5.962099321s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.962129519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 4.898483535s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:02:49 UTC) orbiter warning: context deadline exceeded Duration: 4.898636034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 5.001242385s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:02:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001267742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.......X.X.......X.X.XXX.X.....X.......X...X.............] duration: 5.741699422s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.741746292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.558027479s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:01:46 UTC) orbiter warning: context deadline exceeded Duration: 3.558057096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X.X.......X.X.XXX.X.....X.......X...X............] duration: 5.907944277s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.907970778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.370085637s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:00:49 UTC) orbiter warning: context deadline exceeded Duration: 3.370115182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.......X.X.......X.X.XXX.X.....X.......X...X...........] duration: 6.095428702s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.095457638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.972349197s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:00:31 UTC) orbiter warning: context deadline exceeded Duration: 3.972438999s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)