Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXXX.X.X] duration: 30.00298521s trace: https://jaeger.dev.storj.io/trace/3ed6bc50e8adeef FAIL (2025-12-27 04:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 6T4LFOUKBN7W6PEZRCAW5OAAWWAXUSG6LMH4ESPZY5Y24ZFGYXNQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: S2GAQ63CEN2KJF75CMMOKHVG62YFIMHRXBZLCAFPSWQUWG232N6Q) context deadline exceeded ======LOGS====== ments... Dec 27 04:22:12.661024080 (516h53m12.532107125s) [upload=453510]: Got next segment. Inline: false Dec 27 04:22:12.661062173 (516h53m12.532145208s) [upload=453510]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:22:12.661068925 (516h53m12.53215197s) [upload=453510]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:22:12.904959734 (516h53m12.776042779s) [seg_pos={0 0}, upload=453510]: Begin upload segment... Dec 27 04:22:12.905005661 (516h53m12.776088697s) [seg_pos={0 0}, upload=453510]: Done begin upload segment. Dec 27 04:22:12.905031391 (516h53m12.776114436s) [noise=true , address=145.79.227.38:4978 , piece_id=LDDBUT66SIXE3FJZE3IFBFKPIPFW52GNLS27VPTX5SNUJ3I63YFQ, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.905190043 (516h53m12.776273088s) [noise=true , address=145.79.227.8:4330 , piece_id=KXS2VMM7M2KVIO2TUT4INSR3F2RLVLRLFOFPZJBJOVRA5FZCTLZA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.905622186 (516h53m12.776705221s) [noise=true , address=145.79.227.47:4265 , piece_id=VWWGX7C4Q3QXGQ3EPGQFPDNSLPMUATQTOWOXGH3JPNXDG75BPE5A, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.906258156 (516h53m12.777341202s) [noise=true , address=145.79.227.7:4293 , piece_id=AJ7N2ZGYOESD3H7LBY634BMCG2SIMMFPQBF5WFTH72PUGJWCC54Q, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.906369118 (516h53m12.777452153s) [noise=true , address=77.90.31.16:4736 , piece_id=6T4LFOUKBN7W6PEZRCAW5OAAWWAXUSG6LMH4ESPZY5Y24ZFGYXNQ, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.907077848 (516h53m12.778160893s) [noise=true , address=45.91.116.7:4009 , piece_id=SYOG36B47RGKMGJMOFAGZYPB3IWCRDPSDIC62VUDIZGHH2PEPYXA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.907144705 (516h53m12.77822775s) [noise=true , address=145.79.227.39:4014 , piece_id=ZG6I2SKS73LR2Q46CACKXMYV4BZ4MCHGB7YOVFMGJFVPHPBLZBRA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.907870768 (516h53m12.778953813s) [noise=true , address=145.79.227.29:4943 , piece_id=6ECB77C2YYPEZXZYS4W6RAL573KTRV5HORDWZK2GIF4UJ3XCEIFA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.907950630 (516h53m12.779033665s) [noise=true , address=145.79.227.15:4873 , piece_id=KTR2MVY4CYIBUHFN2TT5ZLWZYN3E3ZXU6FWKYN7G7VY7N22BKI2A, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.908359038 (516h53m12.779442073s) [noise=true , address=45.91.116.10:4083 , piece_id=3U7EDFITULDQLL6WLFPG6RD6NSQBKFGM2IAWDCNKYDXWFI2V6EMQ, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.908904055 (516h53m12.77998709s) [noise=true , address=45.91.116.6:4975 , piece_id=RSTUI5WFDDO3SMOBSLMZ7TOWGNKPKA4XUCZDNMICIJO2A72KYMYA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.909304117 (516h53m12.780387162s) [noise=true , address=145.79.227.13:4802 , piece_id=5VHHQ7WQQZBCYLGRDWQXABZ5OBJFK5U7DEF7GW3S37ZGVN43LDUA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.909660906 (516h53m12.780743951s) [noise=true , address=145.79.227.11:4730 , piece_id=JDM6YUDLEX2VSZ7ZOQXUGBH4BWRZIT7L7MNTHERO4P5SQIXKBRHA, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.910320783 (516h53m12.781403828s) [noise=true , address=145.79.227.52:4447 , piece_id=TO7VRRAMV53D44M5HQPDGPAQCCDMEVLPAVGP4DOXNMALM6AJWC4A, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.910741434 (516h53m12.781824469s) [noise=true , address=77.90.31.10:4519 , piece_id=S2GAQ63CEN2KJF75CMMOKHVG62YFIMHRXBZLCAFPSWQUWG232N6Q, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:12.911195258 (516h53m12.782278303s) [noise=true , address=145.79.227.42:4123 , piece_id=IXFC6XKUOQCVSOPHQZPPXV5FDRYBVY2GP4BESB2JYK2O6ZZXAOLQ, seg_pos={0 0}, upload=453510]: Uploading piece... Dec 27 04:22:14.583015908 (516h53m14.454098953s) [upload=453510]: Next returned nil segment Dec 27 04:22:14.583035495 (516h53m14.45411853s) [upload=453510]: Waiting for error group managing segments... Dec 27 04:22:14.902505838 (516h53m14.773588873s) [noise=true , address=145.79.227.39:4014 , piece_id=ZG6I2SKS73LR2Q46CACKXMYV4BZ4MCHGB7YOVFMGJFVPHPBLZBRA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.902568858 (516h53m14.773651893s) [noise=true , address=145.79.227.52:4447 , piece_id=TO7VRRAMV53D44M5HQPDGPAQCCDMEVLPAVGP4DOXNMALM6AJWC4A, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.903092856 (516h53m14.774175891s) [noise=true , address=145.79.227.15:4873 , piece_id=KTR2MVY4CYIBUHFN2TT5ZLWZYN3E3ZXU6FWKYN7G7VY7N22BKI2A, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.904800618 (516h53m14.775883663s) [noise=true , address=145.79.227.38:4978 , piece_id=LDDBUT66SIXE3FJZE3IFBFKPIPFW52GNLS27VPTX5SNUJ3I63YFQ, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.911493723 (516h53m14.782576758s) [noise=true , address=145.79.227.13:4802 , piece_id=5VHHQ7WQQZBCYLGRDWQXABZ5OBJFK5U7DEF7GW3S37ZGVN43LDUA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.913982121 (516h53m14.785065156s) [noise=true , address=145.79.227.8:4330 , piece_id=KXS2VMM7M2KVIO2TUT4INSR3F2RLVLRLFOFPZJBJOVRA5FZCTLZA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.914815418 (516h53m14.785898453s) [noise=true , address=145.79.227.7:4293 , piece_id=AJ7N2ZGYOESD3H7LBY634BMCG2SIMMFPQBF5WFTH72PUGJWCC54Q, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.916735554 (516h53m14.787818589s) [noise=true , address=145.79.227.29:4943 , piece_id=6ECB77C2YYPEZXZYS4W6RAL573KTRV5HORDWZK2GIF4UJ3XCEIFA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.918527776 (516h53m14.789610811s) [noise=true , address=145.79.227.11:4730 , piece_id=JDM6YUDLEX2VSZ7ZOQXUGBH4BWRZIT7L7MNTHERO4P5SQIXKBRHA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.921902152 (516h53m14.792985197s) [noise=true , address=145.79.227.47:4265 , piece_id=VWWGX7C4Q3QXGQ3EPGQFPDNSLPMUATQTOWOXGH3JPNXDG75BPE5A, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:14.934191082 (516h53m14.805274117s) [noise=true , address=145.79.227.42:4123 , piece_id=IXFC6XKUOQCVSOPHQZPPXV5FDRYBVY2GP4BESB2JYK2O6ZZXAOLQ, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:15.084006054 (516h53m14.955089099s) [noise=true , address=77.90.31.10:4519 , piece_id=S2GAQ63CEN2KJF75CMMOKHVG62YFIMHRXBZLCAFPSWQUWG232N6Q, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:15.094377463 (516h53m14.965460518s) [noise=true , address=77.90.31.16:4736 , piece_id=6T4LFOUKBN7W6PEZRCAW5OAAWWAXUSG6LMH4ESPZY5Y24ZFGYXNQ, seg_pos={0 0}, upload=453510]: Done uploading piece. err: <nil> Dec 27 04:22:15.094406709 (516h53m14.965489754s) [seg_pos={0 0}, upload=453510]: Segment reached optimal threshold of 13 pieces. Dec 27 04:22:15.094727109 (516h53m14.965810154s) [noise=true , address=45.91.116.7:4009 , piece_id=SYOG36B47RGKMGJMOFAGZYPB3IWCRDPSDIC62VUDIZGHH2PEPYXA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 04:22:15.094736157 (516h53m14.965819222s) [noise=true , address=45.91.116.7:4009 , piece_id=SYOG36B47RGKMGJMOFAGZYPB3IWCRDPSDIC62VUDIZGHH2PEPYXA, seg_pos={0 0}, upload=453510]: Long tail cancellation due to optimal threshold Dec 27 04:22:15.094894407 (516h53m14.965977442s) [noise=true , address=45.91.116.10:4083 , piece_id=3U7EDFITULDQLL6WLFPG6RD6NSQBKFGM2IAWDCNKYDXWFI2V6EMQ, seg_pos={0 0}, upload=453510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:40222->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:40222->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40222->45.91.116.10:4083: use of closed network connection Dec 27 04:22:15.094901501 (516h53m14.965984536s) [noise=true , address=45.91.116.10:4083 , piece_id=3U7EDFITULDQLL6WLFPG6RD6NSQBKFGM2IAWDCNKYDXWFI2V6EMQ, seg_pos={0 0}, upload=453510]: Long tail cancellation due to optimal threshold Dec 27 04:22:15.095258631 (516h53m14.966341676s) [noise=true , address=45.91.116.6:4975 , piece_id=RSTUI5WFDDO3SMOBSLMZ7TOWGNKPKA4XUCZDNMICIJO2A72KYMYA, seg_pos={0 0}, upload=453510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:59156->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:59156->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59156->45.91.116.6:4975: use of closed network connection Dec 27 04:22:15.095277076 (516h53m14.966360111s) [noise=true , address=45.91.116.6:4975 , piece_id=RSTUI5WFDDO3SMOBSLMZ7TOWGNKPKA4XUCZDNMICIJO2A72KYMYA, seg_pos={0 0}, upload=453510]: Long tail cancellation due to optimal threshold Dec 27 04:22:15.095305901 (516h53m14.966388936s) [seg_pos={0 0}, upload=453510]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:22:15.095400622 (516h53m14.966483667s) [upload=453510]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:22:15.095420489 (516h53m14.966503524s) [upload=453510]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:22:15.378962069 (516h53m15.250045104s) [upload=453510]: Done uploading segments... Dec 27 04:22:15.378982207 (516h53m15.250065252s) [upload=453510]: Upload finished. err: <nil> Duration: 30.003009887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.....X............X.............................] duration: 47.615715135s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 47.615740774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXXX.X.] duration: 5.90463198s trace: https://jaeger.dev.storj.io/trace/15d99115b370f1a5 FAIL (2025-12-27 04:21:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:34.425879119 (516h51m34.296962154s) [upload=453487]: Starting upload Dec 27 04:20:34.425965903 (516h51m34.297048959s) [upload=453487]: Done starting upload Dec 27 04:20:34.426077536 (516h51m34.297160571s) [upload=453487]: Uploading segments... Dec 27 04:20:34.426478139 (516h51m34.297561184s) [upload=453487]: Got next segment. Inline: false Dec 27 04:20:34.426616543 (516h51m34.297699578s) [upload=453487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:20:34.426626431 (516h51m34.297709476s) [upload=453487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:20:34.676854617 (516h51m34.547937662s) [seg_pos={0 0}, upload=453487]: Begin upload segment... Dec 27 04:20:34.676955799 (516h51m34.548038845s) [seg_pos={0 0}, upload=453487]: Done begin upload segment. Dec 27 04:20:34.677026684 (516h51m34.548109729s) [noise=true , address=145.79.227.45:4229 , piece_id=LRDX6LRTJRCXK6MSR2UNBDFK34BZJQEQ34TVGI6KWYCZYIOM54IQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677073944 (516h51m34.548156979s) [noise=true , address=77.90.31.14:4661 , piece_id=JIFJLCMXTNQA5LSQE5VTI244VGC53NLC7WTXN6ID2IQGNAL7YSMA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677765351 (516h51m34.548848396s) [noise=true , address=145.79.227.51:4411 , piece_id=NMRXYTNPMJXE2XD5A7HXTVHT3HXLLNWAEYPNQWYBZEEEGY4D77AA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677939222 (516h51m34.549022257s) [noise=true , address=145.79.227.52:4445 , piece_id=AMOLGEP43OJ4OPCNDMKAVRHCAKJ5FPFFHRIRXKQ446CIJ2IWB2KQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678295199 (516h51m34.549378234s) [noise=true , address=145.79.227.8:4330 , piece_id=DYGZTSEOW35LV3ZU7JAAARN4RIUIIPWPI3ZHYEZFLEVOOX6PV4FQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678499679 (516h51m34.549582714s) [noise=true , address=145.79.227.4:4185 , piece_id=X6VEMB26QH3Z5PXZHHJIY5SNANW5QECZN2GJ6RNI53CDWRZY5PLA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678857750 (516h51m34.549940785s) [noise=true , address=145.79.227.10:4401 , piece_id=GYUCTHDFHH5F72BLRLR6CRLV3GKVTUY3EF74NR7OYNG5ZQ4R5EZQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678912665 (516h51m34.54999571s) [noise=true , address=45.91.116.11:4117 , piece_id=Q77MWDUACUNU7SHY5RNCGSAYBIZLUYUPJYY3LOE7QL7KHAYLEKPQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679252803 (516h51m34.550335848s) [noise=true , address=45.91.116.10:4083 , piece_id=Y36GSAPMSYC7FQDLI4CXETFWT5BLKUQ6DVRPWBTNSG4PWES3SQTQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679403540 (516h51m34.550486575s) [noise=true , address=45.91.116.8:4047 , piece_id=ZNQ4ZCWVJG5ZWCOKK6E7WD75WFZWIZLXRXEFNYIF3YDBFMNGAUKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679848848 (516h51m34.550931883s) [noise=true , address=145.79.227.42:4121 , piece_id=KZS6MM6NVVNURSGSTLNBXNTXKNAMFCF52RAVQTSORER3XYJWWX4A, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.680385680 (516h51m34.551468725s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.680927172 (516h51m34.552010217s) [noise=true , address=145.79.227.11:4732 , piece_id=3EUMMM4JNIJYRZ6VVXA3CS7HZG5CR7LIXYIXDGTM54WMHYPK2LDA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.681445118 (516h51m34.552528153s) [noise=true , address=145.79.227.48:4302 , piece_id=QZEZMSMWNSXMPOWRCLJR3NVFJP74EH72EX6RJBBY7U77HVIIGGKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.681859287 (516h51m34.552942332s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.684769968 (516h51m34.555853004s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:36.550726846 (516h51m36.421809901s) [upload=453487]: Next returned nil segment Dec 27 04:20:36.550744719 (516h51m36.421827764s) [upload=453487]: Waiting for error group managing segments... Dec 27 04:20:36.856104524 (516h51m36.727187569s) [noise=true , address=145.79.227.11:4732 , piece_id=3EUMMM4JNIJYRZ6VVXA3CS7HZG5CR7LIXYIXDGTM54WMHYPK2LDA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.875771963 (516h51m36.746855008s) [noise=true , address=145.79.227.45:4229 , piece_id=LRDX6LRTJRCXK6MSR2UNBDFK34BZJQEQ34TVGI6KWYCZYIOM54IQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.886425320 (516h51m36.757508365s) [noise=true , address=145.79.227.48:4302 , piece_id=QZEZMSMWNSXMPOWRCLJR3NVFJP74EH72EX6RJBBY7U77HVIIGGKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.894150339 (516h51m36.765233384s) [noise=true , address=145.79.227.8:4330 , piece_id=DYGZTSEOW35LV3ZU7JAAARN4RIUIIPWPI3ZHYEZFLEVOOX6PV4FQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.912259682 (516h51m36.783342727s) [noise=true , address=77.90.31.14:4661 , piece_id=JIFJLCMXTNQA5LSQE5VTI244VGC53NLC7WTXN6ID2IQGNAL7YSMA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.947039328 (516h51m36.818122373s) [noise=true , address=145.79.227.4:4185 , piece_id=X6VEMB26QH3Z5PXZHHJIY5SNANW5QECZN2GJ6RNI53CDWRZY5PLA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.948950858 (516h51m36.820033913s) [noise=true , address=45.91.116.10:4083 , piece_id=Y36GSAPMSYC7FQDLI4CXETFWT5BLKUQ6DVRPWBTNSG4PWES3SQTQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.976831117 (516h51m36.847914162s) [noise=true , address=145.79.227.51:4411 , piece_id=NMRXYTNPMJXE2XD5A7HXTVHT3HXLLNWAEYPNQWYBZEEEGY4D77AA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.046797243 (516h51m36.917880278s) [noise=true , address=45.91.116.11:4117 , piece_id=Q77MWDUACUNU7SHY5RNCGSAYBIZLUYUPJYY3LOE7QL7KHAYLEKPQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.149621547 (516h51m37.020704592s) [noise=true , address=45.91.116.8:4047 , piece_id=ZNQ4ZCWVJG5ZWCOKK6E7WD75WFZWIZLXRXEFNYIF3YDBFMNGAUKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.598736715 (516h51m37.46981975s) [noise=true , address=145.79.227.52:4445 , piece_id=AMOLGEP43OJ4OPCNDMKAVRHCAKJ5FPFFHRIRXKQ446CIJ2IWB2KQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.621699020 (516h51m37.492782055s) [noise=true , address=145.79.227.42:4121 , piece_id=KZS6MM6NVVNURSGSTLNBXNTXKNAMFCF52RAVQTSORER3XYJWWX4A, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.625555634 (516h51m37.496638669s) [noise=true , address=145.79.227.10:4401 , piece_id=GYUCTHDFHH5F72BLRLR6CRLV3GKVTUY3EF74NR7OYNG5ZQ4R5EZQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.625567887 (516h51m37.496650932s) [seg_pos={0 0}, upload=453487]: Segment reached optimal threshold of 13 pieces. Dec 27 04:20:37.625735797 (516h51m37.496818832s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Dec 27 04:20:37.625742309 (516h51m37.496825344s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.625991854 (516h51m37.497074899s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection Dec 27 04:20:37.625997845 (516h51m37.49708089s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.626105470 (516h51m37.497188505s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 27 04:20:37.626113175 (516h51m37.497196221s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.626130157 (516h51m37.497213202s) [seg_pos={0 0}, upload=453487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:20:37.626193568 (516h51m37.497276603s) [upload=453487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:20:37.626198067 (516h51m37.497281102s) [upload=453487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:20:37.919254274 (516h51m37.790337329s) [upload=453487]: Done uploading segments... Dec 27 04:20:37.919278579 (516h51m37.790361624s) [upload=453487]: Upload finished. err: <nil> Duration: 5.904660243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.....X............X............................] duration: 44.185497415s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:21:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 44.185523144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXXX.X] duration: 30.003920716s trace: https://jaeger.dev.storj.io/trace/15d99115b370f1a5 FAIL (2025-12-27 04:21:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:34.425879119 (516h51m34.296962154s) [upload=453487]: Starting upload Dec 27 04:20:34.425965903 (516h51m34.297048959s) [upload=453487]: Done starting upload Dec 27 04:20:34.426077536 (516h51m34.297160571s) [upload=453487]: Uploading segments... Dec 27 04:20:34.426478139 (516h51m34.297561184s) [upload=453487]: Got next segment. Inline: false Dec 27 04:20:34.426616543 (516h51m34.297699578s) [upload=453487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:20:34.426626431 (516h51m34.297709476s) [upload=453487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:20:34.676854617 (516h51m34.547937662s) [seg_pos={0 0}, upload=453487]: Begin upload segment... Dec 27 04:20:34.676955799 (516h51m34.548038845s) [seg_pos={0 0}, upload=453487]: Done begin upload segment. Dec 27 04:20:34.677026684 (516h51m34.548109729s) [noise=true , address=145.79.227.45:4229 , piece_id=LRDX6LRTJRCXK6MSR2UNBDFK34BZJQEQ34TVGI6KWYCZYIOM54IQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677073944 (516h51m34.548156979s) [noise=true , address=77.90.31.14:4661 , piece_id=JIFJLCMXTNQA5LSQE5VTI244VGC53NLC7WTXN6ID2IQGNAL7YSMA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677765351 (516h51m34.548848396s) [noise=true , address=145.79.227.51:4411 , piece_id=NMRXYTNPMJXE2XD5A7HXTVHT3HXLLNWAEYPNQWYBZEEEGY4D77AA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.677939222 (516h51m34.549022257s) [noise=true , address=145.79.227.52:4445 , piece_id=AMOLGEP43OJ4OPCNDMKAVRHCAKJ5FPFFHRIRXKQ446CIJ2IWB2KQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678295199 (516h51m34.549378234s) [noise=true , address=145.79.227.8:4330 , piece_id=DYGZTSEOW35LV3ZU7JAAARN4RIUIIPWPI3ZHYEZFLEVOOX6PV4FQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678499679 (516h51m34.549582714s) [noise=true , address=145.79.227.4:4185 , piece_id=X6VEMB26QH3Z5PXZHHJIY5SNANW5QECZN2GJ6RNI53CDWRZY5PLA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678857750 (516h51m34.549940785s) [noise=true , address=145.79.227.10:4401 , piece_id=GYUCTHDFHH5F72BLRLR6CRLV3GKVTUY3EF74NR7OYNG5ZQ4R5EZQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.678912665 (516h51m34.54999571s) [noise=true , address=45.91.116.11:4117 , piece_id=Q77MWDUACUNU7SHY5RNCGSAYBIZLUYUPJYY3LOE7QL7KHAYLEKPQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679252803 (516h51m34.550335848s) [noise=true , address=45.91.116.10:4083 , piece_id=Y36GSAPMSYC7FQDLI4CXETFWT5BLKUQ6DVRPWBTNSG4PWES3SQTQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679403540 (516h51m34.550486575s) [noise=true , address=45.91.116.8:4047 , piece_id=ZNQ4ZCWVJG5ZWCOKK6E7WD75WFZWIZLXRXEFNYIF3YDBFMNGAUKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.679848848 (516h51m34.550931883s) [noise=true , address=145.79.227.42:4121 , piece_id=KZS6MM6NVVNURSGSTLNBXNTXKNAMFCF52RAVQTSORER3XYJWWX4A, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.680385680 (516h51m34.551468725s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.680927172 (516h51m34.552010217s) [noise=true , address=145.79.227.11:4732 , piece_id=3EUMMM4JNIJYRZ6VVXA3CS7HZG5CR7LIXYIXDGTM54WMHYPK2LDA, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.681445118 (516h51m34.552528153s) [noise=true , address=145.79.227.48:4302 , piece_id=QZEZMSMWNSXMPOWRCLJR3NVFJP74EH72EX6RJBBY7U77HVIIGGKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.681859287 (516h51m34.552942332s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:34.684769968 (516h51m34.555853004s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Uploading piece... Dec 27 04:20:36.550726846 (516h51m36.421809901s) [upload=453487]: Next returned nil segment Dec 27 04:20:36.550744719 (516h51m36.421827764s) [upload=453487]: Waiting for error group managing segments... Dec 27 04:20:36.856104524 (516h51m36.727187569s) [noise=true , address=145.79.227.11:4732 , piece_id=3EUMMM4JNIJYRZ6VVXA3CS7HZG5CR7LIXYIXDGTM54WMHYPK2LDA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.875771963 (516h51m36.746855008s) [noise=true , address=145.79.227.45:4229 , piece_id=LRDX6LRTJRCXK6MSR2UNBDFK34BZJQEQ34TVGI6KWYCZYIOM54IQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.886425320 (516h51m36.757508365s) [noise=true , address=145.79.227.48:4302 , piece_id=QZEZMSMWNSXMPOWRCLJR3NVFJP74EH72EX6RJBBY7U77HVIIGGKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.894150339 (516h51m36.765233384s) [noise=true , address=145.79.227.8:4330 , piece_id=DYGZTSEOW35LV3ZU7JAAARN4RIUIIPWPI3ZHYEZFLEVOOX6PV4FQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.912259682 (516h51m36.783342727s) [noise=true , address=77.90.31.14:4661 , piece_id=JIFJLCMXTNQA5LSQE5VTI244VGC53NLC7WTXN6ID2IQGNAL7YSMA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.947039328 (516h51m36.818122373s) [noise=true , address=145.79.227.4:4185 , piece_id=X6VEMB26QH3Z5PXZHHJIY5SNANW5QECZN2GJ6RNI53CDWRZY5PLA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.948950858 (516h51m36.820033913s) [noise=true , address=45.91.116.10:4083 , piece_id=Y36GSAPMSYC7FQDLI4CXETFWT5BLKUQ6DVRPWBTNSG4PWES3SQTQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:36.976831117 (516h51m36.847914162s) [noise=true , address=145.79.227.51:4411 , piece_id=NMRXYTNPMJXE2XD5A7HXTVHT3HXLLNWAEYPNQWYBZEEEGY4D77AA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.046797243 (516h51m36.917880278s) [noise=true , address=45.91.116.11:4117 , piece_id=Q77MWDUACUNU7SHY5RNCGSAYBIZLUYUPJYY3LOE7QL7KHAYLEKPQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.149621547 (516h51m37.020704592s) [noise=true , address=45.91.116.8:4047 , piece_id=ZNQ4ZCWVJG5ZWCOKK6E7WD75WFZWIZLXRXEFNYIF3YDBFMNGAUKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.598736715 (516h51m37.46981975s) [noise=true , address=145.79.227.52:4445 , piece_id=AMOLGEP43OJ4OPCNDMKAVRHCAKJ5FPFFHRIRXKQ446CIJ2IWB2KQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.621699020 (516h51m37.492782055s) [noise=true , address=145.79.227.42:4121 , piece_id=KZS6MM6NVVNURSGSTLNBXNTXKNAMFCF52RAVQTSORER3XYJWWX4A, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.625555634 (516h51m37.496638669s) [noise=true , address=145.79.227.10:4401 , piece_id=GYUCTHDFHH5F72BLRLR6CRLV3GKVTUY3EF74NR7OYNG5ZQ4R5EZQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: <nil> Dec 27 04:20:37.625567887 (516h51m37.496650932s) [seg_pos={0 0}, upload=453487]: Segment reached optimal threshold of 13 pieces. Dec 27 04:20:37.625735797 (516h51m37.496818832s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Dec 27 04:20:37.625742309 (516h51m37.496825344s) [noise=true , address=145.79.227.20:4439 , piece_id=EHF5QV65E5XZYIE2UGHIOCDMFOP2GG4TDKL3XAV3WHNQAF5K3RVQ, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.625991854 (516h51m37.497074899s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44176->145.79.227.49:4339: use of closed network connection Dec 27 04:20:37.625997845 (516h51m37.49708089s) [noise=true , address=145.79.227.49:4339 , piece_id=CKPRK3MAQHDREOSXNGLTJX7ON5N6XO2FSV4BVO3GUQ7PRQG5ZJKQ, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.626105470 (516h51m37.497188505s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 27 04:20:37.626113175 (516h51m37.497196221s) [noise=true , address=145.79.227.47:4267 , piece_id=AYGJ35KPFJOUEHNAOPSDKSPOJLWLFIFAEXDG2WX5TS2476J4GJYA, seg_pos={0 0}, upload=453487]: Long tail cancellation due to optimal threshold Dec 27 04:20:37.626130157 (516h51m37.497213202s) [seg_pos={0 0}, upload=453487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:20:37.626193568 (516h51m37.497276603s) [upload=453487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:20:37.626198067 (516h51m37.497281102s) [upload=453487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:20:37.919254274 (516h51m37.790337329s) [upload=453487]: Done uploading segments... Dec 27 04:20:37.919278579 (516h51m37.790361624s) [upload=453487]: Upload finished. err: <nil> Duration: 30.003947096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....X............X...........................] duration: 30.455247516s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 30.455277312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXXX.] duration: 6.965517661s trace: https://jaeger.dev.storj.io/trace/117b4bc14d02eedf FAIL (2025-12-27 04:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: 434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q) context deadline exceeded ======LOGS====== Dec 27 04:19:02.916981020 (516h50m2.788064065s) [upload=453466]: Starting upload Dec 27 04:19:02.917067214 (516h50m2.788150269s) [upload=453466]: Done starting upload Dec 27 04:19:02.917198113 (516h50m2.788281168s) [upload=453466]: Uploading segments... Dec 27 04:19:02.917421118 (516h50m2.788504173s) [upload=453466]: Got next segment. Inline: false Dec 27 04:19:02.917499857 (516h50m2.788582902s) [upload=453466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:19:02.917508234 (516h50m2.788591279s) [upload=453466]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:19:03.165224707 (516h50m3.036307742s) [seg_pos={0 0}, upload=453466]: Begin upload segment... Dec 27 04:19:03.165289791 (516h50m3.036372836s) [seg_pos={0 0}, upload=453466]: Done begin upload segment. Dec 27 04:19:03.165330077 (516h50m3.036413132s) [noise=true , address=45.91.116.6:4975 , piece_id=PC4BBGW5CWBVUMWSQ7FLLTVWJVJVOTTYK4Z77JQDZAHCLCQSE37Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.170242742 (516h50m3.041325787s) [noise=true , address=145.79.227.11:4732 , piece_id=AN5NM3WGSRZEQYVAMU5ZK6UMTYB4AZLDDEBO46X5KDX7RVSRQC6A, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.170783140 (516h50m3.041866175s) [noise=true , address=145.79.227.39:4015 , piece_id=XEWZC4KVODLPPJL2B4XEZGXSLGQI3Z3ERYZLMEZBBKXT3YZYBX4Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.172601042 (516h50m3.043684097s) [noise=true , address=77.90.31.10:4520 , piece_id=434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.172627753 (516h50m3.043710798s) [noise=true , address=145.79.227.52:4445 , piece_id=3EODKKV7PKKWQGPNUC4QHM5475Q4CRPV3H7SBIMZWKC6WEGNFGUQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.174276101 (516h50m3.045359146s) [noise=true , address=145.79.227.37:4565 , piece_id=KBTNAAL42NRXCVYJIXLCZZVCK75EOMKXUFTA26TBJTIY2WSTMMDQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.174494919 (516h50m3.045577964s) [noise=true , address=145.79.227.40:4050 , piece_id=VIPJJGNGIGEBO5NE7EOGUPLSTM6BIOEN6RLYVLW5AQX4HEPQX4VQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.175921294 (516h50m3.047004339s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.176117698 (516h50m3.047200743s) [noise=true , address=145.79.227.36:4517 , piece_id=HD7ZSU25Y6CFRPJ525RP7XZI23COBUTIXMQZB3YOPHKFRLNY276A, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.177632212 (516h50m3.048715247s) [noise=true , address=145.79.227.20:4439 , piece_id=QGBFIJYBOTCJQPQYRVKZRH3AOJM4GOXN52BHD3ZUE5AY7SMZK5ZQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.177714809 (516h50m3.048797854s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.179499197 (516h50m3.050582252s) [noise=true , address=145.79.227.31:4799 , piece_id=3U6JIIQJ3OMQPSCOKWFV57HKWYMEDTODXCH3Z6YSGHHUCRKVWXRA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.179944926 (516h50m3.051027981s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.181063576 (516h50m3.052146641s) [noise=true , address=145.79.227.47:4265 , piece_id=6EXZMGC52QPAC4BTINWUPRXYQTQ5EPIWQ55WWVTHCESXINLSDQIQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.182315159 (516h50m3.053398204s) [noise=true , address=145.79.227.17:4970 , piece_id=UEFRHRPIPCPKST62BNY2FVT45UVE2RF6FBUQ6GE4AWHMSVVUC3FA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.184262968 (516h50m3.055346013s) [noise=true , address=45.91.116.8:4048 , piece_id=BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:04.803809899 (516h50m4.674892934s) [upload=453466]: Next returned nil segment Dec 27 04:19:04.803822102 (516h50m4.674905147s) [upload=453466]: Waiting for error group managing segments... Dec 27 04:19:05.185730308 (516h50m5.056813343s) [noise=true , address=145.79.227.11:4732 , piece_id=AN5NM3WGSRZEQYVAMU5ZK6UMTYB4AZLDDEBO46X5KDX7RVSRQC6A, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.185776075 (516h50m5.05685912s) [noise=true , address=145.79.227.31:4799 , piece_id=3U6JIIQJ3OMQPSCOKWFV57HKWYMEDTODXCH3Z6YSGHHUCRKVWXRA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.187117479 (516h50m5.058200514s) [noise=true , address=145.79.227.20:4439 , piece_id=QGBFIJYBOTCJQPQYRVKZRH3AOJM4GOXN52BHD3ZUE5AY7SMZK5ZQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.190188086 (516h50m5.061271131s) [noise=true , address=145.79.227.39:4015 , piece_id=XEWZC4KVODLPPJL2B4XEZGXSLGQI3Z3ERYZLMEZBBKXT3YZYBX4Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.202791214 (516h50m5.07387425s) [noise=true , address=145.79.227.52:4445 , piece_id=3EODKKV7PKKWQGPNUC4QHM5475Q4CRPV3H7SBIMZWKC6WEGNFGUQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.204619005 (516h50m5.07570204s) [noise=true , address=145.79.227.47:4265 , piece_id=6EXZMGC52QPAC4BTINWUPRXYQTQ5EPIWQ55WWVTHCESXINLSDQIQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.205579824 (516h50m5.076662869s) [noise=true , address=145.79.227.36:4517 , piece_id=HD7ZSU25Y6CFRPJ525RP7XZI23COBUTIXMQZB3YOPHKFRLNY276A, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.209613736 (516h50m5.080696771s) [noise=true , address=145.79.227.17:4970 , piece_id=UEFRHRPIPCPKST62BNY2FVT45UVE2RF6FBUQ6GE4AWHMSVVUC3FA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.213300826 (516h50m5.084383871s) [noise=true , address=145.79.227.40:4050 , piece_id=VIPJJGNGIGEBO5NE7EOGUPLSTM6BIOEN6RLYVLW5AQX4HEPQX4VQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.222249326 (516h50m5.093332371s) [noise=true , address=145.79.227.37:4565 , piece_id=KBTNAAL42NRXCVYJIXLCZZVCK75EOMKXUFTA26TBJTIY2WSTMMDQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.431040848 (516h50m5.302123893s) [noise=true , address=45.91.116.6:4975 , piece_id=PC4BBGW5CWBVUMWSQ7FLLTVWJVJVOTTYK4Z77JQDZAHCLCQSE37Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.432389366 (516h50m5.303472401s) [noise=true , address=45.91.116.8:4048 , piece_id=BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.468601340 (516h50m5.339684385s) [noise=true , address=77.90.31.10:4520 , piece_id=434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.468638681 (516h50m5.339721716s) [seg_pos={0 0}, upload=453466]: Segment reached optimal threshold of 13 pieces. Dec 27 04:19:05.468979701 (516h50m5.340062746s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.468990882 (516h50m5.340073927s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469058731 (516h50m5.340141776s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.469076195 (516h50m5.34015924s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469313606 (516h50m5.340396641s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.469321522 (516h50m5.340404557s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469345036 (516h50m5.340428071s) [seg_pos={0 0}, upload=453466]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:19:05.469526111 (516h50m5.340609146s) [upload=453466]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:19:05.469541140 (516h50m5.340624175s) [upload=453466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:19:05.772764466 (516h50m5.643847512s) [upload=453466]: Done uploading segments... Dec 27 04:19:05.772786369 (516h50m5.643869404s) [upload=453466]: Upload finished. err: <nil> Duration: 6.965530066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....X............X..........................] duration: 42.963657534s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:19:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 42.963686961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXXX] duration: 30.002604288s trace: https://jaeger.dev.storj.io/trace/117b4bc14d02eedf FAIL (2025-12-27 04:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: 434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q) context deadline exceeded ======LOGS====== Dec 27 04:19:02.916981020 (516h50m2.788064065s) [upload=453466]: Starting upload Dec 27 04:19:02.917067214 (516h50m2.788150269s) [upload=453466]: Done starting upload Dec 27 04:19:02.917198113 (516h50m2.788281168s) [upload=453466]: Uploading segments... Dec 27 04:19:02.917421118 (516h50m2.788504173s) [upload=453466]: Got next segment. Inline: false Dec 27 04:19:02.917499857 (516h50m2.788582902s) [upload=453466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:19:02.917508234 (516h50m2.788591279s) [upload=453466]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:19:03.165224707 (516h50m3.036307742s) [seg_pos={0 0}, upload=453466]: Begin upload segment... Dec 27 04:19:03.165289791 (516h50m3.036372836s) [seg_pos={0 0}, upload=453466]: Done begin upload segment. Dec 27 04:19:03.165330077 (516h50m3.036413132s) [noise=true , address=45.91.116.6:4975 , piece_id=PC4BBGW5CWBVUMWSQ7FLLTVWJVJVOTTYK4Z77JQDZAHCLCQSE37Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.170242742 (516h50m3.041325787s) [noise=true , address=145.79.227.11:4732 , piece_id=AN5NM3WGSRZEQYVAMU5ZK6UMTYB4AZLDDEBO46X5KDX7RVSRQC6A, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.170783140 (516h50m3.041866175s) [noise=true , address=145.79.227.39:4015 , piece_id=XEWZC4KVODLPPJL2B4XEZGXSLGQI3Z3ERYZLMEZBBKXT3YZYBX4Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.172601042 (516h50m3.043684097s) [noise=true , address=77.90.31.10:4520 , piece_id=434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.172627753 (516h50m3.043710798s) [noise=true , address=145.79.227.52:4445 , piece_id=3EODKKV7PKKWQGPNUC4QHM5475Q4CRPV3H7SBIMZWKC6WEGNFGUQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.174276101 (516h50m3.045359146s) [noise=true , address=145.79.227.37:4565 , piece_id=KBTNAAL42NRXCVYJIXLCZZVCK75EOMKXUFTA26TBJTIY2WSTMMDQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.174494919 (516h50m3.045577964s) [noise=true , address=145.79.227.40:4050 , piece_id=VIPJJGNGIGEBO5NE7EOGUPLSTM6BIOEN6RLYVLW5AQX4HEPQX4VQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.175921294 (516h50m3.047004339s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.176117698 (516h50m3.047200743s) [noise=true , address=145.79.227.36:4517 , piece_id=HD7ZSU25Y6CFRPJ525RP7XZI23COBUTIXMQZB3YOPHKFRLNY276A, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.177632212 (516h50m3.048715247s) [noise=true , address=145.79.227.20:4439 , piece_id=QGBFIJYBOTCJQPQYRVKZRH3AOJM4GOXN52BHD3ZUE5AY7SMZK5ZQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.177714809 (516h50m3.048797854s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.179499197 (516h50m3.050582252s) [noise=true , address=145.79.227.31:4799 , piece_id=3U6JIIQJ3OMQPSCOKWFV57HKWYMEDTODXCH3Z6YSGHHUCRKVWXRA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.179944926 (516h50m3.051027981s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.181063576 (516h50m3.052146641s) [noise=true , address=145.79.227.47:4265 , piece_id=6EXZMGC52QPAC4BTINWUPRXYQTQ5EPIWQ55WWVTHCESXINLSDQIQ, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.182315159 (516h50m3.053398204s) [noise=true , address=145.79.227.17:4970 , piece_id=UEFRHRPIPCPKST62BNY2FVT45UVE2RF6FBUQ6GE4AWHMSVVUC3FA, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:03.184262968 (516h50m3.055346013s) [noise=true , address=45.91.116.8:4048 , piece_id=BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q, seg_pos={0 0}, upload=453466]: Uploading piece... Dec 27 04:19:04.803809899 (516h50m4.674892934s) [upload=453466]: Next returned nil segment Dec 27 04:19:04.803822102 (516h50m4.674905147s) [upload=453466]: Waiting for error group managing segments... Dec 27 04:19:05.185730308 (516h50m5.056813343s) [noise=true , address=145.79.227.11:4732 , piece_id=AN5NM3WGSRZEQYVAMU5ZK6UMTYB4AZLDDEBO46X5KDX7RVSRQC6A, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.185776075 (516h50m5.05685912s) [noise=true , address=145.79.227.31:4799 , piece_id=3U6JIIQJ3OMQPSCOKWFV57HKWYMEDTODXCH3Z6YSGHHUCRKVWXRA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.187117479 (516h50m5.058200514s) [noise=true , address=145.79.227.20:4439 , piece_id=QGBFIJYBOTCJQPQYRVKZRH3AOJM4GOXN52BHD3ZUE5AY7SMZK5ZQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.190188086 (516h50m5.061271131s) [noise=true , address=145.79.227.39:4015 , piece_id=XEWZC4KVODLPPJL2B4XEZGXSLGQI3Z3ERYZLMEZBBKXT3YZYBX4Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.202791214 (516h50m5.07387425s) [noise=true , address=145.79.227.52:4445 , piece_id=3EODKKV7PKKWQGPNUC4QHM5475Q4CRPV3H7SBIMZWKC6WEGNFGUQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.204619005 (516h50m5.07570204s) [noise=true , address=145.79.227.47:4265 , piece_id=6EXZMGC52QPAC4BTINWUPRXYQTQ5EPIWQ55WWVTHCESXINLSDQIQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.205579824 (516h50m5.076662869s) [noise=true , address=145.79.227.36:4517 , piece_id=HD7ZSU25Y6CFRPJ525RP7XZI23COBUTIXMQZB3YOPHKFRLNY276A, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.209613736 (516h50m5.080696771s) [noise=true , address=145.79.227.17:4970 , piece_id=UEFRHRPIPCPKST62BNY2FVT45UVE2RF6FBUQ6GE4AWHMSVVUC3FA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.213300826 (516h50m5.084383871s) [noise=true , address=145.79.227.40:4050 , piece_id=VIPJJGNGIGEBO5NE7EOGUPLSTM6BIOEN6RLYVLW5AQX4HEPQX4VQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.222249326 (516h50m5.093332371s) [noise=true , address=145.79.227.37:4565 , piece_id=KBTNAAL42NRXCVYJIXLCZZVCK75EOMKXUFTA26TBJTIY2WSTMMDQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.431040848 (516h50m5.302123893s) [noise=true , address=45.91.116.6:4975 , piece_id=PC4BBGW5CWBVUMWSQ7FLLTVWJVJVOTTYK4Z77JQDZAHCLCQSE37Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.432389366 (516h50m5.303472401s) [noise=true , address=45.91.116.8:4048 , piece_id=BWTYJD4OKFEAN2JP3GO6JHBL3LLO3RQFTWHOVAZCBOF3GXEVKR7Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.468601340 (516h50m5.339684385s) [noise=true , address=77.90.31.10:4520 , piece_id=434ZCOEVISP4YUHEQUCRRARNISU5XWRCUUWIGZUIKCKGLWPTRBBA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: <nil> Dec 27 04:19:05.468638681 (516h50m5.339721716s) [seg_pos={0 0}, upload=453466]: Segment reached optimal threshold of 13 pieces. Dec 27 04:19:05.468979701 (516h50m5.340062746s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.468990882 (516h50m5.340073927s) [noise=true , address=77.90.31.16:4736 , piece_id=6QTJ53XTDKTCWT7FMCSWFEM2YMY66D6M45UTY4OM25RLA463XLOA, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469058731 (516h50m5.340141776s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.469076195 (516h50m5.34015924s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGBJMBYHEGZYLJMFDRWD26XDWELSBLE7URXDKHKIIJHAZFLUI37Q, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469313606 (516h50m5.340396641s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:19:05.469321522 (516h50m5.340404557s) [noise=true , address=77.90.31.18:4808 , piece_id=NAWIBXTLZ5HVBF2BJQSQXHIA3VGNW4LSYMA733BRXZN6S5GBFEMQ, seg_pos={0 0}, upload=453466]: Long tail cancellation due to optimal threshold Dec 27 04:19:05.469345036 (516h50m5.340428071s) [seg_pos={0 0}, upload=453466]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:19:05.469526111 (516h50m5.340609146s) [upload=453466]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:19:05.469541140 (516h50m5.340624175s) [upload=453466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:19:05.772764466 (516h50m5.643847512s) [upload=453466]: Done uploading segments... Dec 27 04:19:05.772786369 (516h50m5.643869404s) [upload=453466]: Upload finished. err: <nil> Duration: 30.002618295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XXX] duration: 30.004553502s trace: https://jaeger.dev.storj.io/trace/7aeafcb29d2dfb81 FAIL (2025-12-27 04:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:18:32.650154226 (516h49m32.521237271s) [upload=453458]: Starting upload Dec 27 04:18:32.650241983 (516h49m32.521325028s) [upload=453458]: Done starting upload Dec 27 04:18:32.650555560 (516h49m32.521638605s) [upload=453458]: Uploading segments... Dec 27 04:18:32.650675860 (516h49m32.521758905s) [upload=453458]: Got next segment. Inline: false Dec 27 04:18:32.650721206 (516h49m32.521804241s) [upload=453458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:18:32.650727648 (516h49m32.521810693s) [upload=453458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:18:32.896525067 (516h49m32.767608122s) [seg_pos={0 0}, upload=453458]: Begin upload segment... Dec 27 04:18:32.896593457 (516h49m32.767676492s) [seg_pos={0 0}, upload=453458]: Done begin upload segment. Dec 27 04:18:32.896637190 (516h49m32.767720225s) [noise=true , address=45.91.116.9:4262 , piece_id=TOPYHE6J6OOHTVUUBWW4ZKEZR6ITUDU6BD3QZUCVIPMKJKBDYLLA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.896656337 (516h49m32.767739382s) [noise=true , address=145.79.227.49:4339 , piece_id=5YXXUO2LTQKJKAVQJGIHUWCIEK6VGORNHXG4G3WSHJMUK57ADZDA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.896713575 (516h49m32.76779661s) [noise=true , address=145.79.227.2:4113 , piece_id=AJG6ZA7UR6O5FKXQZAAGTZS36HXWQYWJ45UM3RVZWOS2M65M3XYA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.897167309 (516h49m32.768250354s) [noise=true , address=145.79.227.6:4259 , piece_id=QXFJOYAI5W2TNWH3KCEFPMVD2VIUPUQRD32NQDOS5QAC4FT7ZQEQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.897173240 (516h49m32.768256285s) [noise=true , address=145.79.227.46:4602 , piece_id=XDNKV6QBRUHO6BCTNRV4IYDFEV7ASLFIWXPN56TJJ3OZTPE6SLVQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.897519229 (516h49m32.768602264s) [noise=true , address=77.90.31.16:4736 , piece_id=F6L6RQZKGHXHS2QKUDQJWXDHKCUERRIDQV7PPBFUAJMPOYJJ5SGQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.897586587 (516h49m32.768669632s) [noise=true , address=45.91.116.10:4081 , piece_id=YNGI76A2HZPKFXGZF3JDZR2KMXL27GIEFCSV6VMLIN7ILALL3AZA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.897936023 (516h49m32.769019098s) [noise=true , address=77.90.31.11:4554 , piece_id=3YCXKQR5YCNHZYMX4DTDQBKNBEJTKDLJ4N63HID73KRFGYV26LAQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.898418902 (516h49m32.769501937s) [noise=true , address=45.91.116.5:4298 , piece_id=3LFFEJHX67E3XE52SUHIX2DURTBMN5Q64SK6YL23HSPIL5AKVBAQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.898820277 (516h49m32.769903322s) [noise=true , address=145.79.227.16:4911 , piece_id=UPQS2TYMW4EA446CQBTHDWYME7THBTBEKKAIH342AWTMGGHT2DYA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.899228894 (516h49m32.770311929s) [noise=true , address=145.79.227.50:4375 , piece_id=PGFPQWQOHTQ2OJWFHB6XFXDUUZSEQFRFOFQOGVBU2AG3XSBMMIXQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.899572128 (516h49m32.770655163s) [noise=true , address=145.79.227.13:4802 , piece_id=RIPJBL4MCHSJ5VTFMP4QLV7KXGZE52E665KRN7Y4ZE2SLAH7KS5Q, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.900005674 (516h49m32.771088739s) [noise=true , address=45.91.116.8:4047 , piece_id=AUUMCOFAOSN64RQECJ4U4IV4NWOP3WG4IQ3VYPIJ72QSK6PMFBZA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.900487622 (516h49m32.771570657s) [noise=true , address=145.79.227.15:4874 , piece_id=JQUUBEMATR7NI45IFKHCWRY5C5BE7CGDEROJDONS5V3IQDSI7KGQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.900590668 (516h49m32.771673713s) [noise=true , address=45.91.116.14:4192 , piece_id=YY4YIWEETFCIKHWGSPFAQY67YRRNKHRBNWNZ7UR5MRK4ZBUP7VTQ, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:32.901054421 (516h49m32.772137476s) [noise=true , address=145.79.227.7:4293 , piece_id=475CXQ3B7FVECMMAMO7GAD5277BMOVUGMMHFTXI2JIQD6CU55MWA, seg_pos={0 0}, upload=453458]: Uploading piece... Dec 27 04:18:34.597020668 (516h49m34.468103703s) [upload=453458]: Next returned nil segment Dec 27 04:18:34.597035826 (516h49m34.468118872s) [upload=453458]: Waiting for error group managing segments... Dec 27 04:18:34.899278045 (516h49m34.77036109s) [noise=true , address=145.79.227.50:4375 , piece_id=PGFPQWQOHTQ2OJWFHB6XFXDUUZSEQFRFOFQOGVBU2AG3XSBMMIXQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.906229972 (516h49m34.777313017s) [noise=true , address=145.79.227.6:4259 , piece_id=QXFJOYAI5W2TNWH3KCEFPMVD2VIUPUQRD32NQDOS5QAC4FT7ZQEQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.911834905 (516h49m34.78291795s) [noise=true , address=145.79.227.15:4874 , piece_id=JQUUBEMATR7NI45IFKHCWRY5C5BE7CGDEROJDONS5V3IQDSI7KGQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.924771349 (516h49m34.795854394s) [noise=true , address=145.79.227.49:4339 , piece_id=5YXXUO2LTQKJKAVQJGIHUWCIEK6VGORNHXG4G3WSHJMUK57ADZDA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.926104337 (516h49m34.797187372s) [noise=true , address=145.79.227.16:4911 , piece_id=UPQS2TYMW4EA446CQBTHDWYME7THBTBEKKAIH342AWTMGGHT2DYA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.927751373 (516h49m34.798834408s) [noise=true , address=145.79.227.13:4802 , piece_id=RIPJBL4MCHSJ5VTFMP4QLV7KXGZE52E665KRN7Y4ZE2SLAH7KS5Q, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.935086210 (516h49m34.806169245s) [noise=true , address=145.79.227.46:4602 , piece_id=XDNKV6QBRUHO6BCTNRV4IYDFEV7ASLFIWXPN56TJJ3OZTPE6SLVQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.935914697 (516h49m34.806997732s) [noise=true , address=145.79.227.2:4113 , piece_id=AJG6ZA7UR6O5FKXQZAAGTZS36HXWQYWJ45UM3RVZWOS2M65M3XYA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:34.950226249 (516h49m34.821309294s) [noise=true , address=145.79.227.7:4293 , piece_id=475CXQ3B7FVECMMAMO7GAD5277BMOVUGMMHFTXI2JIQD6CU55MWA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:35.128844337 (516h49m34.999927382s) [noise=true , address=45.91.116.8:4047 , piece_id=AUUMCOFAOSN64RQECJ4U4IV4NWOP3WG4IQ3VYPIJ72QSK6PMFBZA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:35.161238640 (516h49m35.032321675s) [noise=true , address=45.91.116.14:4192 , piece_id=YY4YIWEETFCIKHWGSPFAQY67YRRNKHRBNWNZ7UR5MRK4ZBUP7VTQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:35.182531225 (516h49m35.05361426s) [noise=true , address=45.91.116.5:4298 , piece_id=3LFFEJHX67E3XE52SUHIX2DURTBMN5Q64SK6YL23HSPIL5AKVBAQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:35.264040181 (516h49m35.135123216s) [noise=true , address=77.90.31.16:4736 , piece_id=F6L6RQZKGHXHS2QKUDQJWXDHKCUERRIDQV7PPBFUAJMPOYJJ5SGQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: <nil> Dec 27 04:18:35.264065419 (516h49m35.135148464s) [seg_pos={0 0}, upload=453458]: Segment reached optimal threshold of 13 pieces. Dec 27 04:18:35.264274037 (516h49m35.135357072s) [noise=true , address=45.91.116.9:4262 , piece_id=TOPYHE6J6OOHTVUUBWW4ZKEZR6ITUDU6BD3QZUCVIPMKJKBDYLLA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 04:18:35.264279858 (516h49m35.135362903s) [noise=true , address=45.91.116.9:4262 , piece_id=TOPYHE6J6OOHTVUUBWW4ZKEZR6ITUDU6BD3QZUCVIPMKJKBDYLLA, seg_pos={0 0}, upload=453458]: Long tail cancellation due to optimal threshold Dec 27 04:18:35.264512751 (516h49m35.135595786s) [noise=true , address=45.91.116.10:4081 , piece_id=YNGI76A2HZPKFXGZF3JDZR2KMXL27GIEFCSV6VMLIN7ILALL3AZA, seg_pos={0 0}, upload=453458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:47920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:47920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47920->45.91.116.10:4081: use of closed network connection Dec 27 04:18:35.264519534 (516h49m35.135602579s) [noise=true , address=45.91.116.10:4081 , piece_id=YNGI76A2HZPKFXGZF3JDZR2KMXL27GIEFCSV6VMLIN7ILALL3AZA, seg_pos={0 0}, upload=453458]: Long tail cancellation due to optimal threshold Dec 27 04:18:35.264638230 (516h49m35.135721265s) [noise=true , address=77.90.31.11:4554 , piece_id=3YCXKQR5YCNHZYMX4DTDQBKNBEJTKDLJ4N63HID73KRFGYV26LAQ, seg_pos={0 0}, upload=453458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 04:18:35.264658198 (516h49m35.135741233s) [noise=true , address=77.90.31.11:4554 , piece_id=3YCXKQR5YCNHZYMX4DTDQBKNBEJTKDLJ4N63HID73KRFGYV26LAQ, seg_pos={0 0}, upload=453458]: Long tail cancellation due to optimal threshold Dec 27 04:18:35.264720457 (516h49m35.135803502s) [seg_pos={0 0}, upload=453458]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:18:35.264800138 (516h49m35.135883173s) [upload=453458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:18:35.264805238 (516h49m35.135888273s) [upload=453458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:18:35.553446718 (516h49m35.424529753s) [upload=453458]: Done uploading segments... Dec 27 04:18:35.553469101 (516h49m35.424552136s) [upload=453458]: Upload finished. err: <nil> Duration: 30.004635007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....X............X.........................] duration: 40.511824651s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:18:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 40.511842635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....XX] duration: 30.003668443s trace: https://jaeger.dev.storj.io/trace/6aeb47a5137a0461 FAIL (2025-12-27 04:18:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8m52.316290904s) [upload=453450]: Done starting upload Dec 27 04:17:52.445292149 (516h48m52.316375194s) [upload=453450]: Uploading segments... Dec 27 04:17:52.445453958 (516h48m52.316537003s) [upload=453450]: Got next segment. Inline: false Dec 27 04:17:52.445506617 (516h48m52.316589672s) [upload=453450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:17:52.445513131 (516h48m52.316596176s) [upload=453450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:17:52.952846209 (516h48m52.823929254s) [seg_pos={0 0}, upload=453450]: Begin upload segment... Dec 27 04:17:52.952915121 (516h48m52.823998166s) [seg_pos={0 0}, upload=453450]: Done begin upload segment. Dec 27 04:17:52.952953504 (516h48m52.824036549s) [noise=true , address=77.90.31.10:4517 , piece_id=XMIX4Y4VI2OK35BUXSR7RJNKJBCHHAWIPHGQFWHN7URCUSPFQZQA, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.953090605 (516h48m52.82417364s) [noise=true , address=145.79.227.49:4338 , piece_id=ZGQKGKS2YJR4W5NF4GHDUSZGKUH3CZZ3JFMQQTTZTPOP77T3REZQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.954151114 (516h48m52.825234159s) [noise=true , address=145.79.227.48:4302 , piece_id=XIE5ERZYQWB5A3ITKST4HYA2X3JSWJ22Q37VRU6IPY7ZCHK2GLUQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.954837621 (516h48m52.825920666s) [noise=true , address=145.79.227.38:4978 , piece_id=JIJBWLYYJGQEMG73MZP2MWL5UJFCBT3B2QSINNIRTEVLDMAV46CQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.955362721 (516h48m52.826445776s) [noise=true , address=145.79.227.12:4765 , piece_id=IBTRGXKJS7S6Z6WX4QMZYPAORKRGKIXNYAYD75CHNPGPYAGEBF3Q, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.955855730 (516h48m52.826938775s) [noise=true , address=77.90.31.11:4554 , piece_id=6CGCPGKKVXNTD7HLVDEVXQHRYO6OEJSNZCVVHDHO567EI2TFK7PA, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.957091724 (516h48m52.828174759s) [noise=true , address=145.79.227.52:4447 , piece_id=GNYGOEHA3LS7WMOZXKGHJHOUDSMKIZEL4PVUWPKXTTWIT7XW2EOQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.957522453 (516h48m52.828605498s) [noise=true , address=145.79.227.16:4911 , piece_id=MRJDPFDL4JIBK35PR7AZPVWQBEINOJBH3SKK3V3B72NC4JT4EV2A, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.958994035 (516h48m52.83007707s) [noise=true , address=77.90.31.12:4590 , piece_id=IOWMU5XQTWNQFVA4N5EK6JHE37DDKGMEQIHCW72SSP3SXHBAQ6BQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.961912693 (516h48m52.832995728s) [noise=true , address=145.79.227.15:4875 , piece_id=52PAPAQFRSCXTDYAEWV2OZQBPXVEK5KMXZL7IABF75ILL4W32U2Q, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.962923347 (516h48m52.834006393s) [noise=true , address=145.79.227.42:4122 , piece_id=DJTB466BGVG3HRRS3VAUEDUWSPB76AGL3WSDWW5UTJH5PCCKIE2Q, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.967060546 (516h48m52.838143581s) [noise=true , address=145.79.227.11:4732 , piece_id=GDGALJDHGONEWS6RBUR4MKAXN7TYB5ZSYANFYQV3BG6H7P4V5DLQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.967701556 (516h48m52.838784601s) [noise=true , address=145.79.227.17:4972 , piece_id=USK36UVDEXCAUAXVUX3DBYTISLET4CK4XKRRVTI4UGAJDDCYNXUQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.970889456 (516h48m52.841972492s) [noise=true , address=45.91.116.4:4937 , piece_id=OXVEXT24RHJJLKKHHR6M3OMYVKSA5OPXF7ELY5TYRR4XNTMV5GCQ, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.972836814 (516h48m52.843919859s) [noise=true , address=45.91.116.6:4974 , piece_id=GI4V5U36VDMF5GZOS2TWYYZYXE77NVFDTQBME767RGDCBUYYUADA, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:52.979636171 (516h48m52.850719217s) [noise=true , address=145.79.227.50:4373 , piece_id=FXAIUF4J7YTO3P6NJCNDI7LUKSU46EB2DA3GSRQYNYE3L3ONF42A, seg_pos={0 0}, upload=453450]: Uploading piece... Dec 27 04:17:54.804551427 (516h48m54.675634462s) [upload=453450]: Next returned nil segment Dec 27 04:17:54.804576604 (516h48m54.67565965s) [upload=453450]: Waiting for error group managing segments... Dec 27 04:17:55.142656254 (516h48m55.013739309s) [noise=true , address=145.79.227.42:4122 , piece_id=DJTB466BGVG3HRRS3VAUEDUWSPB76AGL3WSDWW5UTJH5PCCKIE2Q, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.150496823 (516h48m55.021579868s) [noise=true , address=145.79.227.15:4875 , piece_id=52PAPAQFRSCXTDYAEWV2OZQBPXVEK5KMXZL7IABF75ILL4W32U2Q, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.156319892 (516h48m55.027402927s) [noise=true , address=145.79.227.17:4972 , piece_id=USK36UVDEXCAUAXVUX3DBYTISLET4CK4XKRRVTI4UGAJDDCYNXUQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.158680236 (516h48m55.029763271s) [noise=true , address=77.90.31.12:4590 , piece_id=IOWMU5XQTWNQFVA4N5EK6JHE37DDKGMEQIHCW72SSP3SXHBAQ6BQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.161625494 (516h48m55.032708529s) [noise=true , address=145.79.227.52:4447 , piece_id=GNYGOEHA3LS7WMOZXKGHJHOUDSMKIZEL4PVUWPKXTTWIT7XW2EOQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.183728823 (516h48m55.054811858s) [noise=true , address=145.79.227.16:4911 , piece_id=MRJDPFDL4JIBK35PR7AZPVWQBEINOJBH3SKK3V3B72NC4JT4EV2A, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.251255353 (516h48m55.122338398s) [noise=true , address=145.79.227.11:4732 , piece_id=GDGALJDHGONEWS6RBUR4MKAXN7TYB5ZSYANFYQV3BG6H7P4V5DLQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.253477184 (516h48m55.124560219s) [noise=true , address=45.91.116.6:4974 , piece_id=GI4V5U36VDMF5GZOS2TWYYZYXE77NVFDTQBME767RGDCBUYYUADA, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.264897891 (516h48m55.135980926s) [noise=true , address=77.90.31.10:4517 , piece_id=XMIX4Y4VI2OK35BUXSR7RJNKJBCHHAWIPHGQFWHN7URCUSPFQZQA, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.311259732 (516h48m55.182342767s) [noise=true , address=45.91.116.4:4937 , piece_id=OXVEXT24RHJJLKKHHR6M3OMYVKSA5OPXF7ELY5TYRR4XNTMV5GCQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.354228572 (516h48m55.225311607s) [noise=true , address=77.90.31.11:4554 , piece_id=6CGCPGKKVXNTD7HLVDEVXQHRYO6OEJSNZCVVHDHO567EI2TFK7PA, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.455345995 (516h48m55.32642903s) [noise=true , address=145.79.227.50:4373 , piece_id=FXAIUF4J7YTO3P6NJCNDI7LUKSU46EB2DA3GSRQYNYE3L3ONF42A, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.481526107 (516h48m55.352609162s) [noise=true , address=145.79.227.12:4765 , piece_id=IBTRGXKJS7S6Z6WX4QMZYPAORKRGKIXNYAYD75CHNPGPYAGEBF3Q, seg_pos={0 0}, upload=453450]: Done uploading piece. err: <nil> Dec 27 04:17:55.481550433 (516h48m55.352633478s) [seg_pos={0 0}, upload=453450]: Segment reached optimal threshold of 13 pieces. Dec 27 04:17:55.481761044 (516h48m55.352844079s) [noise=true , address=145.79.227.49:4338 , piece_id=ZGQKGKS2YJR4W5NF4GHDUSZGKUH3CZZ3JFMQQTTZTPOP77T3REZQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:50412->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:50412->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50412->145.79.227.49:4338: use of closed network connection Dec 27 04:17:55.481769310 (516h48m55.352852355s) [noise=true , address=145.79.227.49:4338 , piece_id=ZGQKGKS2YJR4W5NF4GHDUSZGKUH3CZZ3JFMQQTTZTPOP77T3REZQ, seg_pos={0 0}, upload=453450]: Long tail cancellation due to optimal threshold Dec 27 04:17:55.481786152 (516h48m55.352869197s) [noise=true , address=145.79.227.48:4302 , piece_id=XIE5ERZYQWB5A3ITKST4HYA2X3JSWJ22Q37VRU6IPY7ZCHK2GLUQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 27 04:17:55.481794889 (516h48m55.352877924s) [noise=true , address=145.79.227.48:4302 , piece_id=XIE5ERZYQWB5A3ITKST4HYA2X3JSWJ22Q37VRU6IPY7ZCHK2GLUQ, seg_pos={0 0}, upload=453450]: Long tail cancellation due to optimal threshold Dec 27 04:17:55.482070383 (516h48m55.353153418s) [noise=true , address=145.79.227.38:4978 , piece_id=JIJBWLYYJGQEMG73MZP2MWL5UJFCBT3B2QSINNIRTEVLDMAV46CQ, seg_pos={0 0}, upload=453450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: read tcp 188.166.225.170:49002->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 27 04:17:55.482082105 (516h48m55.35316515s) [noise=true , address=145.79.227.38:4978 , piece_id=JIJBWLYYJGQEMG73MZP2MWL5UJFCBT3B2QSINNIRTEVLDMAV46CQ, seg_pos={0 0}, upload=453450]: Long tail cancellation due to optimal threshold Dec 27 04:17:55.482101082 (516h48m55.353184117s) [seg_pos={0 0}, upload=453450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:17:55.482192135 (516h48m55.35327518s) [upload=453450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:17:55.482197916 (516h48m55.353280961s) [upload=453450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:17:56.529790272 (516h48m56.400873317s) [upload=453450]: Done uploading segments... Dec 27 04:17:56.529846809 (516h48m56.400929844s) [upload=453450]: Upload finished. err: <nil> Duration: 30.00368254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X............X........................] duration: 52.567166657s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 52.567194029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....X] duration: 30.003210453s trace: https://jaeger.dev.storj.io/trace/40a7accb282d98a8 FAIL (2025-12-27 04:17:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: TTMCZB4JR47WO2ZDSDPFORGNCN3WKZQBRZO4SM3Z23ZWWFEGBIZA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: LX6HPHSFB64RFA4H5OFMEBMRF5P7JKAM4A2UFRTOH4RNHGPJDXZA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: LM6YXTEKYY6CZGYTC5NKGER4CEF67CYCZ6YDQJWEGQFEFYNDP7BA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: VK357Z5OD7QKR5KNG2GSYPEG5DW5YZ2RZBI7COIUIWJKAVIAS44Q) context deadline exceeded; piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: K3PAH2ZSAQLSVZOA3RPSDCD35LATEEFQT3PH7SQYAG5PTJWZ23IQ) context deadline exceeded ======LOGS====== Dec 27 04:17:25.457327420 (516h48m25.328410495s) [upload=453441]: Starting upload Dec 27 04:17:25.457419936 (516h48m25.328502981s) [upload=453441]: Done starting upload Dec 27 04:17:25.457675643 (516h48m25.328758688s) [upload=453441]: Uploading segments... Dec 27 04:17:25.457767829 (516h48m25.328850874s) [upload=453441]: Got next segment. Inline: false Dec 27 04:17:25.457834505 (516h48m25.32891755s) [upload=453441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:17:25.457840637 (516h48m25.328923682s) [upload=453441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:17:26.260799521 (516h48m26.131882566s) [seg_pos={0 0}, upload=453441]: Begin upload segment... Dec 27 04:17:26.260865126 (516h48m26.131948171s) [seg_pos={0 0}, upload=453441]: Done begin upload segment. Dec 27 04:17:26.260926704 (516h48m26.132009739s) [noise=true , address=45.91.116.12:4228 , piece_id=VK357Z5OD7QKR5KNG2GSYPEG5DW5YZ2RZBI7COIUIWJKAVIAS44Q, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.261789867 (516h48m26.132872912s) [noise=true , address=145.79.227.33:4871 , piece_id=K3PAH2ZSAQLSVZOA3RPSDCD35LATEEFQT3PH7SQYAG5PTJWZ23IQ, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.262775885 (516h48m26.13385893s) [noise=true , address=145.79.227.38:4977 , piece_id=TTMCZB4JR47WO2ZDSDPFORGNCN3WKZQBRZO4SM3Z23ZWWFEGBIZA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.263298570 (516h48m26.134381615s) [noise=true , address=45.91.116.4:4939 , piece_id=JB4GKXBDX52QVRF3HGYQTAPYWGKQNT3AX7NKPBWAYMJC727ICM2A, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.263716265 (516h48m26.13479931s) [noise=true , address=145.79.227.43:4158 , piece_id=LX6HPHSFB64RFA4H5OFMEBMRF5P7JKAM4A2UFRTOH4RNHGPJDXZA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.264177303 (516h48m26.135260348s) [noise=true , address=45.91.116.14:4189 , piece_id=GEDKCUWIJ76E5BGRXKSJHMRF2CB62FAP4F4NOSH5T3EPPNPATRYQ, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.264606931 (516h48m26.135689966s) [noise=true , address=45.91.116.5:4297 , piece_id=RT4DLRWYO2SE7AX36C6CECQ27BVCR5YOJQ4KJLJ77ORSNUT5UZKA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.265079190 (516h48m26.136162236s) [noise=true , address=77.90.31.10:4520 , piece_id=TRCZPAWEY7UJ65MUNDR445TZYY7XOSR6FQD6K6AQSINNQC3HHM3Q, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.265493760 (516h48m26.136576805s) [noise=true , address=145.79.227.40:4049 , piece_id=S6DLC6FHCC2HZILOL3S44ODKYRR24NAXMFXBAOJUAASHCXDXZ72Q, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.267286694 (516h48m26.138369739s) [noise=true , address=77.90.31.16:4735 , piece_id=LM6YXTEKYY6CZGYTC5NKGER4CEF67CYCZ6YDQJWEGQFEFYNDP7BA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.268353776 (516h48m26.139436821s) [noise=true , address=145.79.227.17:4972 , piece_id=7JWCZZVWZWXYOLQ54ICT3JSTIMJODEFFZ6D743XIYOPWWFQNQ3LQ, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.270685997 (516h48m26.141769042s) [noise=true , address=145.79.227.42:4122 , piece_id=OVSVOEHGA5DUH3GIVF6M4SDLIX242GEXF4EVZM5P4EWKDEPE7L7A, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.272160344 (516h48m26.143243389s) [noise=true , address=145.79.227.11:4730 , piece_id=NZSAD7TVO4FVAZ57CAYDZHEAZ5WOHPOHEL33G5FUBCGKBVJDFQQQ, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.273434350 (516h48m26.144517395s) [noise=true , address=77.90.31.13:4627 , piece_id=6GDOOSINYALFJH4L2RZXKQQJKVFTNLXDGZXHMEIVTL5724EVVAMA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.276467716 (516h48m26.147550781s) [noise=true , address=145.79.227.2:4115 , piece_id=V3B3523M5KZOA7JGYXNLYPZGRTXJCC5YP6WZLCOEXJDXHTWXN7QQ, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:26.279108735 (516h48m26.15019178s) [noise=true , address=145.79.227.51:4409 , piece_id=C2STDVNTBCBEPI2GTWWN6JGVI3EMDHTKC5DQLXUTSCQ3KR56XVIA, seg_pos={0 0}, upload=453441]: Uploading piece... Dec 27 04:17:28.037983719 (516h48m27.909066764s) [upload=453441]: Next returned nil segment Dec 27 04:17:28.038002455 (516h48m27.9090855s) [upload=453441]: Waiting for error group managing segments... Dec 27 04:17:28.481542177 (516h48m28.352625212s) [noise=true , address=145.79.227.38:4977 , piece_id=TTMCZB4JR47WO2ZDSDPFORGNCN3WKZQBRZO4SM3Z23ZWWFEGBIZA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.488651284 (516h48m28.359734329s) [noise=true , address=145.79.227.43:4158 , piece_id=LX6HPHSFB64RFA4H5OFMEBMRF5P7JKAM4A2UFRTOH4RNHGPJDXZA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.493112408 (516h48m28.364195463s) [noise=true , address=77.90.31.13:4627 , piece_id=6GDOOSINYALFJH4L2RZXKQQJKVFTNLXDGZXHMEIVTL5724EVVAMA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.547846701 (516h48m28.418929746s) [noise=true , address=77.90.31.10:4520 , piece_id=TRCZPAWEY7UJ65MUNDR445TZYY7XOSR6FQD6K6AQSINNQC3HHM3Q, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.554729147 (516h48m28.425812192s) [noise=true , address=45.91.116.12:4228 , piece_id=VK357Z5OD7QKR5KNG2GSYPEG5DW5YZ2RZBI7COIUIWJKAVIAS44Q, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.570218662 (516h48m28.441301707s) [noise=true , address=45.91.116.4:4939 , piece_id=JB4GKXBDX52QVRF3HGYQTAPYWGKQNT3AX7NKPBWAYMJC727ICM2A, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.634859939 (516h48m28.505942974s) [noise=true , address=45.91.116.5:4297 , piece_id=RT4DLRWYO2SE7AX36C6CECQ27BVCR5YOJQ4KJLJ77ORSNUT5UZKA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.751416061 (516h48m28.622499106s) [noise=true , address=145.79.227.11:4730 , piece_id=NZSAD7TVO4FVAZ57CAYDZHEAZ5WOHPOHEL33G5FUBCGKBVJDFQQQ, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.770425558 (516h48m28.641508593s) [noise=true , address=77.90.31.16:4735 , piece_id=LM6YXTEKYY6CZGYTC5NKGER4CEF67CYCZ6YDQJWEGQFEFYNDP7BA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:28.785391815 (516h48m28.65647486s) [noise=true , address=145.79.227.42:4122 , piece_id=OVSVOEHGA5DUH3GIVF6M4SDLIX242GEXF4EVZM5P4EWKDEPE7L7A, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:29.180579307 (516h48m29.051662342s) [noise=true , address=45.91.116.14:4189 , piece_id=GEDKCUWIJ76E5BGRXKSJHMRF2CB62FAP4F4NOSH5T3EPPNPATRYQ, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:29.557579196 (516h48m29.428662241s) [noise=true , address=145.79.227.2:4115 , piece_id=V3B3523M5KZOA7JGYXNLYPZGRTXJCC5YP6WZLCOEXJDXHTWXN7QQ, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:29.858945637 (516h48m29.730028682s) [noise=true , address=145.79.227.33:4871 , piece_id=K3PAH2ZSAQLSVZOA3RPSDCD35LATEEFQT3PH7SQYAG5PTJWZ23IQ, seg_pos={0 0}, upload=453441]: Done uploading piece. err: <nil> Dec 27 04:17:29.858970043 (516h48m29.730053088s) [seg_pos={0 0}, upload=453441]: Segment reached optimal threshold of 13 pieces. Dec 27 04:17:29.859289361 (516h48m29.730372416s) [noise=true , address=145.79.227.17:4972 , piece_id=7JWCZZVWZWXYOLQ54ICT3JSTIMJODEFFZ6D743XIYOPWWFQNQ3LQ, seg_pos={0 0}, upload=453441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 27 04:17:29.859299771 (516h48m29.730382806s) [noise=true , address=145.79.227.17:4972 , piece_id=7JWCZZVWZWXYOLQ54ICT3JSTIMJODEFFZ6D743XIYOPWWFQNQ3LQ, seg_pos={0 0}, upload=453441]: Long tail cancellation due to optimal threshold Dec 27 04:17:29.859313898 (516h48m29.730396943s) [noise=true , address=145.79.227.51:4409 , piece_id=C2STDVNTBCBEPI2GTWWN6JGVI3EMDHTKC5DQLXUTSCQ3KR56XVIA, seg_pos={0 0}, upload=453441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 27 04:17:29.859333074 (516h48m29.730416119s) [noise=true , address=145.79.227.51:4409 , piece_id=C2STDVNTBCBEPI2GTWWN6JGVI3EMDHTKC5DQLXUTSCQ3KR56XVIA, seg_pos={0 0}, upload=453441]: Long tail cancellation due to optimal threshold Dec 27 04:17:29.859549978 (516h48m29.730633023s) [noise=true , address=145.79.227.40:4049 , piece_id=S6DLC6FHCC2HZILOL3S44ODKYRR24NAXMFXBAOJUAASHCXDXZ72Q, seg_pos={0 0}, upload=453441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 04:17:29.859560908 (516h48m29.730643953s) [noise=true , address=145.79.227.40:4049 , piece_id=S6DLC6FHCC2HZILOL3S44ODKYRR24NAXMFXBAOJUAASHCXDXZ72Q, seg_pos={0 0}, upload=453441]: Long tail cancellation due to optimal threshold Dec 27 04:17:29.859589663 (516h48m29.730672698s) [seg_pos={0 0}, upload=453441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:17:29.859723438 (516h48m29.730806473s) [upload=453441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:17:29.859731553 (516h48m29.730814609s) [upload=453441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:17:30.951853892 (516h48m30.822936937s) [upload=453441]: Done uploading segments... Dec 27 04:17:30.951882376 (516h48m30.822965411s) [upload=453441]: Upload finished. err: <nil> Duration: 30.003223277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X....] duration: 12.683654483s trace: https://jaeger.dev.storj.io/trace/509987a6c9e33216 FAIL (2025-12-27 04:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA) context deadline exceeded ======LOGS====== g segments... Dec 27 04:13:55.491404409 (516h44m55.362487454s) [upload=453394]: Got next segment. Inline: false Dec 27 04:13:55.491447361 (516h44m55.362530396s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:13:55.491452941 (516h44m55.362535996s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:13:55.736713727 (516h44m55.607796763s) [seg_pos={0 0}, upload=453394]: Begin upload segment... Dec 27 04:13:55.736785825 (516h44m55.60786887s) [seg_pos={0 0}, upload=453394]: Done begin upload segment. Dec 27 04:13:55.736852602 (516h44m55.607935647s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.736880175 (516h44m55.60796321s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737419191 (516h44m55.608502226s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737898373 (516h44m55.608981408s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738315918 (516h44m55.609398963s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738858422 (516h44m55.609941467s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739327575 (516h44m55.61041062s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739829029 (516h44m55.610912074s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741282798 (516h44m55.612365843s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741712627 (516h44m55.612795662s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.742830926 (516h44m55.613913971s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.743513685 (516h44m55.61459673s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.744186778 (516h44m55.615269823s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.745654933 (516h44m55.616737978s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.747617911 (516h44m55.618700956s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.748845408 (516h44m55.619928453s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:57.466595935 (516h44m57.33767898s) [upload=453394]: Next returned nil segment Dec 27 04:13:57.466613970 (516h44m57.337697015s) [upload=453394]: Waiting for error group managing segments... Dec 27 04:13:57.750532747 (516h44m57.621615782s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.751377856 (516h44m57.622460891s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.754822034 (516h44m57.625905069s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.758701000 (516h44m57.629784045s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772089104 (516h44m57.643172139s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772568908 (516h44m57.643651943s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.776172689 (516h44m57.647255724s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.926307169 (516h44m57.797390205s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.935745782 (516h44m57.806828817s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.946008454 (516h44m57.817091499s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.967985251 (516h44m57.839068287s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.973023254 (516h44m57.844106289s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982282435 (516h44m57.85336548s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982309687 (516h44m57.853392722s) [seg_pos={0 0}, upload=453394]: Segment reached optimal threshold of 13 pieces. Dec 27 04:13:57.982614688 (516h44m57.853697733s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:13:57.982624386 (516h44m57.853707431s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982765124 (516h44m57.853848169s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection Dec 27 04:13:57.982795061 (516h44m57.853878097s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982952422 (516h44m57.854035457s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection Dec 27 04:13:57.982963262 (516h44m57.854046297s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982984964 (516h44m57.854067999s) [seg_pos={0 0}, upload=453394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:13:57.983090204 (516h44m57.854173249s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:13:57.983096997 (516h44m57.854180042s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:13:58.269816237 (516h44m58.140899292s) [upload=453394]: Done uploading segments... Dec 27 04:13:58.269839992 (516h44m58.140923047s) [upload=453394]: Upload finished. err: <nil> Duration: 12.683684439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X............X.......................] duration: 33.771498416s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 33.7715254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X...] duration: 6.027643162s trace: https://jaeger.dev.storj.io/trace/509987a6c9e33216 FAIL (2025-12-27 04:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA) context deadline exceeded ======LOGS====== g segments... Dec 27 04:13:55.491404409 (516h44m55.362487454s) [upload=453394]: Got next segment. Inline: false Dec 27 04:13:55.491447361 (516h44m55.362530396s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:13:55.491452941 (516h44m55.362535996s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:13:55.736713727 (516h44m55.607796763s) [seg_pos={0 0}, upload=453394]: Begin upload segment... Dec 27 04:13:55.736785825 (516h44m55.60786887s) [seg_pos={0 0}, upload=453394]: Done begin upload segment. Dec 27 04:13:55.736852602 (516h44m55.607935647s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.736880175 (516h44m55.60796321s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737419191 (516h44m55.608502226s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737898373 (516h44m55.608981408s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738315918 (516h44m55.609398963s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738858422 (516h44m55.609941467s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739327575 (516h44m55.61041062s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739829029 (516h44m55.610912074s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741282798 (516h44m55.612365843s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741712627 (516h44m55.612795662s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.742830926 (516h44m55.613913971s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.743513685 (516h44m55.61459673s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.744186778 (516h44m55.615269823s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.745654933 (516h44m55.616737978s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.747617911 (516h44m55.618700956s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.748845408 (516h44m55.619928453s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:57.466595935 (516h44m57.33767898s) [upload=453394]: Next returned nil segment Dec 27 04:13:57.466613970 (516h44m57.337697015s) [upload=453394]: Waiting for error group managing segments... Dec 27 04:13:57.750532747 (516h44m57.621615782s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.751377856 (516h44m57.622460891s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.754822034 (516h44m57.625905069s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.758701000 (516h44m57.629784045s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772089104 (516h44m57.643172139s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772568908 (516h44m57.643651943s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.776172689 (516h44m57.647255724s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.926307169 (516h44m57.797390205s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.935745782 (516h44m57.806828817s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.946008454 (516h44m57.817091499s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.967985251 (516h44m57.839068287s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.973023254 (516h44m57.844106289s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982282435 (516h44m57.85336548s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982309687 (516h44m57.853392722s) [seg_pos={0 0}, upload=453394]: Segment reached optimal threshold of 13 pieces. Dec 27 04:13:57.982614688 (516h44m57.853697733s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:13:57.982624386 (516h44m57.853707431s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982765124 (516h44m57.853848169s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection Dec 27 04:13:57.982795061 (516h44m57.853878097s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982952422 (516h44m57.854035457s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection Dec 27 04:13:57.982963262 (516h44m57.854046297s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982984964 (516h44m57.854067999s) [seg_pos={0 0}, upload=453394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:13:57.983090204 (516h44m57.854173249s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:13:57.983096997 (516h44m57.854180042s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:13:58.269816237 (516h44m58.140899292s) [upload=453394]: Done uploading segments... Dec 27 04:13:58.269839992 (516h44m58.140923047s) [upload=453394]: Upload finished. err: <nil> Duration: 6.027670523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X............X......................] duration: 29.291814758s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 29.291839665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X..] duration: 6.194129684s trace: https://jaeger.dev.storj.io/trace/509987a6c9e33216 FAIL (2025-12-27 04:15:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA) context deadline exceeded ======LOGS====== g segments... Dec 27 04:13:55.491404409 (516h44m55.362487454s) [upload=453394]: Got next segment. Inline: false Dec 27 04:13:55.491447361 (516h44m55.362530396s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:13:55.491452941 (516h44m55.362535996s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:13:55.736713727 (516h44m55.607796763s) [seg_pos={0 0}, upload=453394]: Begin upload segment... Dec 27 04:13:55.736785825 (516h44m55.60786887s) [seg_pos={0 0}, upload=453394]: Done begin upload segment. Dec 27 04:13:55.736852602 (516h44m55.607935647s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.736880175 (516h44m55.60796321s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737419191 (516h44m55.608502226s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737898373 (516h44m55.608981408s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738315918 (516h44m55.609398963s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738858422 (516h44m55.609941467s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739327575 (516h44m55.61041062s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739829029 (516h44m55.610912074s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741282798 (516h44m55.612365843s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741712627 (516h44m55.612795662s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.742830926 (516h44m55.613913971s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.743513685 (516h44m55.61459673s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.744186778 (516h44m55.615269823s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.745654933 (516h44m55.616737978s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.747617911 (516h44m55.618700956s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.748845408 (516h44m55.619928453s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:57.466595935 (516h44m57.33767898s) [upload=453394]: Next returned nil segment Dec 27 04:13:57.466613970 (516h44m57.337697015s) [upload=453394]: Waiting for error group managing segments... Dec 27 04:13:57.750532747 (516h44m57.621615782s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.751377856 (516h44m57.622460891s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.754822034 (516h44m57.625905069s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.758701000 (516h44m57.629784045s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772089104 (516h44m57.643172139s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772568908 (516h44m57.643651943s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.776172689 (516h44m57.647255724s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.926307169 (516h44m57.797390205s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.935745782 (516h44m57.806828817s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.946008454 (516h44m57.817091499s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.967985251 (516h44m57.839068287s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.973023254 (516h44m57.844106289s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982282435 (516h44m57.85336548s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982309687 (516h44m57.853392722s) [seg_pos={0 0}, upload=453394]: Segment reached optimal threshold of 13 pieces. Dec 27 04:13:57.982614688 (516h44m57.853697733s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:13:57.982624386 (516h44m57.853707431s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982765124 (516h44m57.853848169s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection Dec 27 04:13:57.982795061 (516h44m57.853878097s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982952422 (516h44m57.854035457s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection Dec 27 04:13:57.982963262 (516h44m57.854046297s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982984964 (516h44m57.854067999s) [seg_pos={0 0}, upload=453394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:13:57.983090204 (516h44m57.854173249s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:13:57.983096997 (516h44m57.854180042s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:13:58.269816237 (516h44m58.140899292s) [upload=453394]: Done uploading segments... Dec 27 04:13:58.269839992 (516h44m58.140923047s) [upload=453394]: Upload finished. err: <nil> Duration: 6.194154901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X.] duration: 5.89829153s trace: https://jaeger.dev.storj.io/trace/509987a6c9e33216 FAIL (2025-12-27 04:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA) context deadline exceeded ======LOGS====== g segments... Dec 27 04:13:55.491404409 (516h44m55.362487454s) [upload=453394]: Got next segment. Inline: false Dec 27 04:13:55.491447361 (516h44m55.362530396s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:13:55.491452941 (516h44m55.362535996s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:13:55.736713727 (516h44m55.607796763s) [seg_pos={0 0}, upload=453394]: Begin upload segment... Dec 27 04:13:55.736785825 (516h44m55.60786887s) [seg_pos={0 0}, upload=453394]: Done begin upload segment. Dec 27 04:13:55.736852602 (516h44m55.607935647s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.736880175 (516h44m55.60796321s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737419191 (516h44m55.608502226s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737898373 (516h44m55.608981408s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738315918 (516h44m55.609398963s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738858422 (516h44m55.609941467s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739327575 (516h44m55.61041062s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739829029 (516h44m55.610912074s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741282798 (516h44m55.612365843s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741712627 (516h44m55.612795662s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.742830926 (516h44m55.613913971s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.743513685 (516h44m55.61459673s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.744186778 (516h44m55.615269823s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.745654933 (516h44m55.616737978s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.747617911 (516h44m55.618700956s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.748845408 (516h44m55.619928453s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:57.466595935 (516h44m57.33767898s) [upload=453394]: Next returned nil segment Dec 27 04:13:57.466613970 (516h44m57.337697015s) [upload=453394]: Waiting for error group managing segments... Dec 27 04:13:57.750532747 (516h44m57.621615782s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.751377856 (516h44m57.622460891s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.754822034 (516h44m57.625905069s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.758701000 (516h44m57.629784045s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772089104 (516h44m57.643172139s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772568908 (516h44m57.643651943s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.776172689 (516h44m57.647255724s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.926307169 (516h44m57.797390205s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.935745782 (516h44m57.806828817s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.946008454 (516h44m57.817091499s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.967985251 (516h44m57.839068287s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.973023254 (516h44m57.844106289s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982282435 (516h44m57.85336548s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982309687 (516h44m57.853392722s) [seg_pos={0 0}, upload=453394]: Segment reached optimal threshold of 13 pieces. Dec 27 04:13:57.982614688 (516h44m57.853697733s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:13:57.982624386 (516h44m57.853707431s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982765124 (516h44m57.853848169s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection Dec 27 04:13:57.982795061 (516h44m57.853878097s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982952422 (516h44m57.854035457s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection Dec 27 04:13:57.982963262 (516h44m57.854046297s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982984964 (516h44m57.854067999s) [seg_pos={0 0}, upload=453394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:13:57.983090204 (516h44m57.854173249s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:13:57.983096997 (516h44m57.854180042s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:13:58.269816237 (516h44m58.140899292s) [upload=453394]: Done uploading segments... Dec 27 04:13:58.269839992 (516h44m58.140923047s) [upload=453394]: Upload finished. err: <nil> Duration: 5.89832352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....X............X.....................] duration: 33.996221808s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 33.996240554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..X] duration: 30.003436686s trace: https://jaeger.dev.storj.io/trace/509987a6c9e33216 FAIL (2025-12-27 04:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA) context deadline exceeded ======LOGS====== g segments... Dec 27 04:13:55.491404409 (516h44m55.362487454s) [upload=453394]: Got next segment. Inline: false Dec 27 04:13:55.491447361 (516h44m55.362530396s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:13:55.491452941 (516h44m55.362535996s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:13:55.736713727 (516h44m55.607796763s) [seg_pos={0 0}, upload=453394]: Begin upload segment... Dec 27 04:13:55.736785825 (516h44m55.60786887s) [seg_pos={0 0}, upload=453394]: Done begin upload segment. Dec 27 04:13:55.736852602 (516h44m55.607935647s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.736880175 (516h44m55.60796321s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737419191 (516h44m55.608502226s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.737898373 (516h44m55.608981408s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738315918 (516h44m55.609398963s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.738858422 (516h44m55.609941467s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739327575 (516h44m55.61041062s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.739829029 (516h44m55.610912074s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741282798 (516h44m55.612365843s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.741712627 (516h44m55.612795662s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.742830926 (516h44m55.613913971s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.743513685 (516h44m55.61459673s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.744186778 (516h44m55.615269823s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.745654933 (516h44m55.616737978s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.747617911 (516h44m55.618700956s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:55.748845408 (516h44m55.619928453s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Uploading piece... Dec 27 04:13:57.466595935 (516h44m57.33767898s) [upload=453394]: Next returned nil segment Dec 27 04:13:57.466613970 (516h44m57.337697015s) [upload=453394]: Waiting for error group managing segments... Dec 27 04:13:57.750532747 (516h44m57.621615782s) [noise=true , address=145.79.227.45:4231 , piece_id=ODKHVY4Y5TVKN3IO4HQOHWBJOPANQ22RUNDIRGSAL4IO7LSYUESQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.751377856 (516h44m57.622460891s) [noise=true , address=145.79.227.48:4302 , piece_id=INHHEBYC5WFQRNGD4L6HYEVFEKC4BRVLFZJCBS4CA4HZSZCPTWQQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.754822034 (516h44m57.625905069s) [noise=true , address=145.79.227.15:4875 , piece_id=O6RBAFHMRN66J4IYNWDVJWCJIUHTSYJMSHYO7JGTZQHYU33U7KBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.758701000 (516h44m57.629784045s) [noise=true , address=145.79.227.5:4223 , piece_id=I7FYRM635QA3H3LYRG7QOPB4WYU6JQF36DHCPOPYPJPODN7X2VYA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772089104 (516h44m57.643172139s) [noise=true , address=145.79.227.7:4293 , piece_id=V2IM3TGUWEONXUNCGL3BXXH7NT2WV53E6SRU4E53UPAB6KCYG4BA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.772568908 (516h44m57.643651943s) [noise=true , address=145.79.227.36:4517 , piece_id=63J7ZVXWEP2PTVBJDSN7IWQ6R7VETGK4NXXQI2T4WQAIRI5MTVXQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.776172689 (516h44m57.647255724s) [noise=true , address=145.79.227.41:4085 , piece_id=HD4CK7KEG2KRV5QRVKEEXMST4MHNR7DPWUD5MR6F2MC64TOCADKA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.926307169 (516h44m57.797390205s) [noise=true , address=77.90.31.15:4700 , piece_id=B2W7FH2XVPA3ZSXUTCETPYOD2Q5USYSMWOYLSIVGNPYSJ4KLVPGA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.935745782 (516h44m57.806828817s) [noise=true , address=77.90.31.16:4736 , piece_id=WXQ3Q7MR7T333UKUXS4YZKFNRQLAIFKXKZTWH3EMDTCKM3NHULVA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.946008454 (516h44m57.817091499s) [noise=true , address=77.90.31.13:4627 , piece_id=WAKMTI7XN2T6REJ2QEE4GK6MXKCLBRXSZ2OEMGGD5USUU3XJEYMA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.967985251 (516h44m57.839068287s) [noise=true , address=45.91.116.12:4227 , piece_id=63QFT62ESEHZHALBUM4UZBVOJYSRZZJNRFLCADNSC2NCF2ITV3OQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.973023254 (516h44m57.844106289s) [noise=true , address=145.79.227.11:4730 , piece_id=D6HQBB3AEP5SBGPCNW5LNSBHD5T4WYLFOSWUXVCLCZ2RSGXBSVBQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982282435 (516h44m57.85336548s) [noise=true , address=45.91.116.13:4156 , piece_id=DIXOHC4HGX66IHSCJFUKT2VRNL5VVSJFR7Q5C4I6YBWJGC73W6MA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: <nil> Dec 27 04:13:57.982309687 (516h44m57.853392722s) [seg_pos={0 0}, upload=453394]: Segment reached optimal threshold of 13 pieces. Dec 27 04:13:57.982614688 (516h44m57.853697733s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 04:13:57.982624386 (516h44m57.853707431s) [noise=true , address=77.90.31.18:4808 , piece_id=Y66DX36LYWKY2X3Y4HMOGX3YYDLXLGLF7AWCJ44W32QM7DGOPPHA, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982765124 (516h44m57.853848169s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38928->45.91.116.6:4976: use of closed network connection Dec 27 04:13:57.982795061 (516h44m57.853878097s) [noise=true , address=45.91.116.6:4976 , piece_id=HLZ4KCIEGFPDSMDJZ2Q3XHMSSGYPNMQUDBWSL67HIT2IE3LG2WWQ, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982952422 (516h44m57.854035457s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55546->77.90.31.17:4770: use of closed network connection Dec 27 04:13:57.982963262 (516h44m57.854046297s) [noise=true , address=77.90.31.17:4770 , piece_id=V5BUDFJTAF3IIQT5W4RSAZSSVNOFUNEOWKU6AJ22HGHCOBC2PQ5A, seg_pos={0 0}, upload=453394]: Long tail cancellation due to optimal threshold Dec 27 04:13:57.982984964 (516h44m57.854067999s) [seg_pos={0 0}, upload=453394]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:13:57.983090204 (516h44m57.854173249s) [upload=453394]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:13:57.983096997 (516h44m57.854180042s) [upload=453394]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:13:58.269816237 (516h44m58.140899292s) [upload=453394]: Done uploading segments... Dec 27 04:13:58.269839992 (516h44m58.140923047s) [upload=453394]: Upload finished. err: <nil> Duration: 30.003459568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....X............X....................] duration: 28.756006969s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:14:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 28.75603916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....X............X...................] duration: 38.979029407s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 38.979058781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX..] duration: 7.889301562s trace: https://jaeger.dev.storj.io/trace/1b9a3171bea6dafc FAIL (2025-12-27 04:12:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA) context deadline exceeded ======LOGS====== Dec 27 04:11:39.783140620 (516h42m39.654223655s) [upload=453365]: Starting upload Dec 27 04:11:39.783219089 (516h42m39.654302124s) [upload=453365]: Done starting upload Dec 27 04:11:39.783485377 (516h42m39.654568422s) [upload=453365]: Uploading segments... Dec 27 04:11:39.783554318 (516h42m39.654637373s) [upload=453365]: Got next segment. Inline: false Dec 27 04:11:39.783589755 (516h42m39.6546728s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:11:39.783594775 (516h42m39.65467782s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:11:40.031763979 (516h42m39.902847024s) [seg_pos={0 0}, upload=453365]: Begin upload segment... Dec 27 04:11:40.031817560 (516h42m39.902900605s) [seg_pos={0 0}, upload=453365]: Done begin upload segment. Dec 27 04:11:40.031851625 (516h42m39.90293466s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.031861655 (516h42m39.9029447s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032355355 (516h42m39.90343839s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032694259 (516h42m39.903777294s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032990193 (516h42m39.904073228s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.033948387 (516h42m39.905031432s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.034507282 (516h42m39.905590317s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035095181 (516h42m39.906178226s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035986288 (516h42m39.907069323s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.036975351 (516h42m39.908058386s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.039859473 (516h42m39.910942518s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.040611706 (516h42m39.911694741s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.041081310 (516h42m39.912164345s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044133402 (516h42m39.915216437s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044596333 (516h42m39.915679368s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.045653507 (516h42m39.916736552s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:41.755727304 (516h42m41.626810349s) [upload=453365]: Next returned nil segment Dec 27 04:11:41.755738895 (516h42m41.62682194s) [upload=453365]: Waiting for error group managing segments... Dec 27 04:11:42.039936574 (516h42m41.911019609s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.045651716 (516h42m41.916734761s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.058598148 (516h42m41.929681194s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.059843580 (516h42m41.930926615s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.066604113 (516h42m41.937687148s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.074633713 (516h42m41.945716758s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.081160601 (516h42m41.952243636s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.200309510 (516h42m42.071392546s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.299909615 (516h42m42.17099265s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.311975130 (516h42m42.183058195s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.315478100 (516h42m42.186561145s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.320855238 (516h42m42.191938274s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328379766 (516h42m42.199462801s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328406578 (516h42m42.199489623s) [seg_pos={0 0}, upload=453365]: Segment reached optimal threshold of 13 pieces. Dec 27 04:11:42.328814875 (516h42m42.19989791s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection Dec 27 04:11:42.328831366 (516h42m42.199914401s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.328850864 (516h42m42.199933909s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.328884738 (516h42m42.199967773s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329219907 (516h42m42.200302962s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.329249102 (516h42m42.200332147s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329283327 (516h42m42.200366372s) [seg_pos={0 0}, upload=453365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:11:42.329398978 (516h42m42.200482023s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:11:42.329407283 (516h42m42.200490328s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:11:42.651993800 (516h42m42.523076855s) [upload=453365]: Done uploading segments... Dec 27 04:11:42.652014840 (516h42m42.523097875s) [upload=453365]: Upload finished. err: <nil> Duration: 7.889317562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX.] duration: 6.240041325s trace: https://jaeger.dev.storj.io/trace/1b9a3171bea6dafc FAIL (2025-12-27 04:12:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA) context deadline exceeded ======LOGS====== Dec 27 04:11:39.783140620 (516h42m39.654223655s) [upload=453365]: Starting upload Dec 27 04:11:39.783219089 (516h42m39.654302124s) [upload=453365]: Done starting upload Dec 27 04:11:39.783485377 (516h42m39.654568422s) [upload=453365]: Uploading segments... Dec 27 04:11:39.783554318 (516h42m39.654637373s) [upload=453365]: Got next segment. Inline: false Dec 27 04:11:39.783589755 (516h42m39.6546728s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:11:39.783594775 (516h42m39.65467782s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:11:40.031763979 (516h42m39.902847024s) [seg_pos={0 0}, upload=453365]: Begin upload segment... Dec 27 04:11:40.031817560 (516h42m39.902900605s) [seg_pos={0 0}, upload=453365]: Done begin upload segment. Dec 27 04:11:40.031851625 (516h42m39.90293466s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.031861655 (516h42m39.9029447s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032355355 (516h42m39.90343839s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032694259 (516h42m39.903777294s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032990193 (516h42m39.904073228s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.033948387 (516h42m39.905031432s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.034507282 (516h42m39.905590317s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035095181 (516h42m39.906178226s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035986288 (516h42m39.907069323s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.036975351 (516h42m39.908058386s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.039859473 (516h42m39.910942518s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.040611706 (516h42m39.911694741s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.041081310 (516h42m39.912164345s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044133402 (516h42m39.915216437s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044596333 (516h42m39.915679368s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.045653507 (516h42m39.916736552s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:41.755727304 (516h42m41.626810349s) [upload=453365]: Next returned nil segment Dec 27 04:11:41.755738895 (516h42m41.62682194s) [upload=453365]: Waiting for error group managing segments... Dec 27 04:11:42.039936574 (516h42m41.911019609s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.045651716 (516h42m41.916734761s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.058598148 (516h42m41.929681194s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.059843580 (516h42m41.930926615s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.066604113 (516h42m41.937687148s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.074633713 (516h42m41.945716758s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.081160601 (516h42m41.952243636s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.200309510 (516h42m42.071392546s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.299909615 (516h42m42.17099265s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.311975130 (516h42m42.183058195s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.315478100 (516h42m42.186561145s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.320855238 (516h42m42.191938274s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328379766 (516h42m42.199462801s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328406578 (516h42m42.199489623s) [seg_pos={0 0}, upload=453365]: Segment reached optimal threshold of 13 pieces. Dec 27 04:11:42.328814875 (516h42m42.19989791s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection Dec 27 04:11:42.328831366 (516h42m42.199914401s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.328850864 (516h42m42.199933909s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.328884738 (516h42m42.199967773s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329219907 (516h42m42.200302962s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.329249102 (516h42m42.200332147s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329283327 (516h42m42.200366372s) [seg_pos={0 0}, upload=453365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:11:42.329398978 (516h42m42.200482023s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:11:42.329407283 (516h42m42.200490328s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:11:42.651993800 (516h42m42.523076855s) [upload=453365]: Done uploading segments... Dec 27 04:11:42.652014840 (516h42m42.523097875s) [upload=453365]: Upload finished. err: <nil> Duration: 6.240104885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.XX] duration: 30.003817351s trace: https://jaeger.dev.storj.io/trace/1b9a3171bea6dafc FAIL (2025-12-27 04:12:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA) context deadline exceeded ======LOGS====== Dec 27 04:11:39.783140620 (516h42m39.654223655s) [upload=453365]: Starting upload Dec 27 04:11:39.783219089 (516h42m39.654302124s) [upload=453365]: Done starting upload Dec 27 04:11:39.783485377 (516h42m39.654568422s) [upload=453365]: Uploading segments... Dec 27 04:11:39.783554318 (516h42m39.654637373s) [upload=453365]: Got next segment. Inline: false Dec 27 04:11:39.783589755 (516h42m39.6546728s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:11:39.783594775 (516h42m39.65467782s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:11:40.031763979 (516h42m39.902847024s) [seg_pos={0 0}, upload=453365]: Begin upload segment... Dec 27 04:11:40.031817560 (516h42m39.902900605s) [seg_pos={0 0}, upload=453365]: Done begin upload segment. Dec 27 04:11:40.031851625 (516h42m39.90293466s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.031861655 (516h42m39.9029447s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032355355 (516h42m39.90343839s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032694259 (516h42m39.903777294s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.032990193 (516h42m39.904073228s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.033948387 (516h42m39.905031432s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.034507282 (516h42m39.905590317s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035095181 (516h42m39.906178226s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.035986288 (516h42m39.907069323s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.036975351 (516h42m39.908058386s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.039859473 (516h42m39.910942518s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.040611706 (516h42m39.911694741s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.041081310 (516h42m39.912164345s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044133402 (516h42m39.915216437s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.044596333 (516h42m39.915679368s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:40.045653507 (516h42m39.916736552s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Uploading piece... Dec 27 04:11:41.755727304 (516h42m41.626810349s) [upload=453365]: Next returned nil segment Dec 27 04:11:41.755738895 (516h42m41.62682194s) [upload=453365]: Waiting for error group managing segments... Dec 27 04:11:42.039936574 (516h42m41.911019609s) [noise=true , address=145.79.227.4:4185 , piece_id=LUFI4HLQHLFIYMQ4M4OAHGJHOPME4LGNOQAM3ICQD4N7KAFGWVOA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.045651716 (516h42m41.916734761s) [noise=true , address=145.79.227.45:4231 , piece_id=E3OYJK2GSUOOSVVWWZLDAQPKPX2K6KYJF5IZ5JG46IT7IGELLGAQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.058598148 (516h42m41.929681194s) [noise=true , address=145.79.227.6:4257 , piece_id=PGD7VUNVYXEW5WMRER3TZ7ZOSYDA3KUZLBZCG6S5AGJ55EKDA4VA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.059843580 (516h42m41.930926615s) [noise=true , address=145.79.227.41:4085 , piece_id=QIMBXX2A7Q3BOPLDCMNXEO3EJG4J45EFNNVWWB24CGTH7H6KHZFQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.066604113 (516h42m41.937687148s) [noise=true , address=145.79.227.31:4797 , piece_id=GZMD67ZLCTLAFIAUPRB7QCQA267EW7AHQY7YXT54BAD2GJNEFR7Q, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.074633713 (516h42m41.945716758s) [noise=true , address=145.79.227.51:4410 , piece_id=OORNNA5JBZTEDNL77233PCMEFYUHKXTUHMMHOBOAAXYEUKYJTGZQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.081160601 (516h42m41.952243636s) [noise=true , address=145.79.227.39:4013 , piece_id=6BLRD7M7GQ6TFCGCBX2LDJRKYOTBC4YIM6EWQ4PJXAIN4ERMR2HA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.200309510 (516h42m42.071392546s) [noise=true , address=145.79.227.29:4942 , piece_id=EVNGRWWJBTZ74GZ7PSNN4AZMH57KBW3LIKE7KJN6OQGETNNZIUYQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.299909615 (516h42m42.17099265s) [noise=true , address=77.90.31.10:4518 , piece_id=M7LDCKT5ZDLOG6XIKSHTUOHMGP6TXSGA2EGTKLY4HRDHXBDXFJSQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.311975130 (516h42m42.183058195s) [noise=true , address=45.91.116.5:4299 , piece_id=WBSVGW6556PPWKNB3P3MSMTLRILAPM3XRMXUWIFJQGIE5HHTZ7MQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.315478100 (516h42m42.186561145s) [noise=true , address=45.91.116.11:4119 , piece_id=YEQQOKJ42ZGYGD4F6K4E7PF6AOFEZ6H5TYEFCW36GXO2O76H2DDQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.320855238 (516h42m42.191938274s) [noise=true , address=77.90.31.17:4770 , piece_id=V4NDW2KWDWJ62R26JJNORQVYRIF27LGETT2DT5IDUV2OPVG7N5CA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328379766 (516h42m42.199462801s) [noise=true , address=77.90.31.12:4589 , piece_id=SWGLBNZENKBPJR4DHNJLMG3EFIXCUMY7EPNAUSTPQLAYZTMJMLMA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: <nil> Dec 27 04:11:42.328406578 (516h42m42.199489623s) [seg_pos={0 0}, upload=453365]: Segment reached optimal threshold of 13 pieces. Dec 27 04:11:42.328814875 (516h42m42.19989791s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47772->45.91.116.6:4976: use of closed network connection Dec 27 04:11:42.328831366 (516h42m42.199914401s) [noise=true , address=45.91.116.6:4976 , piece_id=P3GNTLFIFJZXZT23O4SQ4AQFCJN6ZXADWZUOLN6YMO6UDMSJTXEQ, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.328850864 (516h42m42.199933909s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.328884738 (516h42m42.199967773s) [noise=true , address=45.91.116.12:4227 , piece_id=MX2FYAMNWACI34V7UNNOIFRGJIACNVWYYCMKDZGPA7I7FOK5CVFA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329219907 (516h42m42.200302962s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 04:11:42.329249102 (516h42m42.200332147s) [noise=true , address=45.91.116.14:4189 , piece_id=SEZCUVNCW4C7NPTNBGJHHGXA2CZ5P6DVPR6XGP5BEOT6LXNYDTHA, seg_pos={0 0}, upload=453365]: Long tail cancellation due to optimal threshold Dec 27 04:11:42.329283327 (516h42m42.200366372s) [seg_pos={0 0}, upload=453365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:11:42.329398978 (516h42m42.200482023s) [upload=453365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:11:42.329407283 (516h42m42.200490328s) [upload=453365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:11:42.651993800 (516h42m42.523076855s) [upload=453365]: Done uploading segments... Dec 27 04:11:42.652014840 (516h42m42.523097875s) [upload=453365]: Upload finished. err: <nil> Duration: 30.003840776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.....X............X..................] duration: 30.668860305s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:11:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 30.668880895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.....X............X.................] duration: 51.397906906s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:11:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 51.39794138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.X] duration: 30.003302977s trace: https://jaeger.dev.storj.io/trace/3ab7bc8a4a2d8ae1 FAIL (2025-12-27 04:10:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: NEWG6KKOJ7AFFMOMMGHDZG3NB4W5RTHV3EKMFANZMIN3QOZN7AZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 43A2CNSKWJE25ADECKKNMOY7ENGZN2TPXCA5KLJERDCUBTF2BYDQ) context deadline exceeded ======LOGS====== 7 04:10:24.809170047 (516h41m24.680253092s) [upload=453350]: Starting upload Dec 27 04:10:24.809312719 (516h41m24.680395764s) [upload=453350]: Done starting upload Dec 27 04:10:24.809463235 (516h41m24.680546281s) [upload=453350]: Uploading segments... Dec 27 04:10:24.809684988 (516h41m24.680768023s) [upload=453350]: Got next segment. Inline: false Dec 27 04:10:24.809778276 (516h41m24.680861321s) [upload=453350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:10:24.809786612 (516h41m24.680869647s) [upload=453350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:10:25.063681390 (516h41m24.934764435s) [seg_pos={0 0}, upload=453350]: Begin upload segment... Dec 27 04:10:25.063743988 (516h41m24.934827023s) [seg_pos={0 0}, upload=453350]: Done begin upload segment. Dec 27 04:10:25.063859198 (516h41m24.934942233s) [noise=true , address=145.79.227.34:4905 , piece_id=7RHZ3MMIHPSNQMJENCX46ZOBGBQNAXHIY4YSIEMW56ZHBBFSIBEA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.063889615 (516h41m24.93497266s) [noise=true , address=145.79.227.9:4366 , piece_id=B2Q4JV3CCJPVHH6DAR7QRD7GXKUMD4J6P7E244WNHYEGF3NCQIAQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.064552909 (516h41m24.935635944s) [noise=true , address=145.79.227.3:4151 , piece_id=BLC4CWLUHGINXQ6UO54NVKVKXHP36OFBNP2H6JZ3BCOY3XSFLEEA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.065916856 (516h41m24.936999891s) [noise=true , address=45.91.116.14:4190 , piece_id=43A2CNSKWJE25ADECKKNMOY7ENGZN2TPXCA5KLJERDCUBTF2BYDQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.066313421 (516h41m24.937396456s) [noise=true , address=145.79.227.37:4567 , piece_id=EP67HNPCONTAOOF6JMKQO36ZO2J34EDLTEGQ5GRNZBUFKDWB5N2Q, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.066809134 (516h41m24.93789217s) [noise=true , address=145.79.227.38:4979 , piece_id=V5GM6VCP5YJSTUPAX57PGDQBANHGDONBG44WJNRKBI2CSRJVFXSA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.067196502 (516h41m24.938279537s) [noise=true , address=145.79.227.42:4122 , piece_id=LCH33YDWNLGGRBSMTRQDIY7IQSGIYXCKK2K6JA2DSTQQJXRGREFQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.067595582 (516h41m24.938678617s) [noise=true , address=145.79.227.13:4802 , piece_id=EOXM35Q45TKYNVLLQBWJCFAMDVPNCA3CZ7MPUHFJOJO32HJWWHSQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.068105384 (516h41m24.939188419s) [noise=true , address=145.79.227.15:4873 , piece_id=FIE76IFA4IMMGPVSMU4Z7B7TYLVUHWJMDAU5U5VBIO2WWWA7NBNQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.068537576 (516h41m24.939620611s) [noise=true , address=145.79.227.41:4086 , piece_id=DXKOQ7MW3NI6LHN5MB72R6LB6G77NIEIMVFR7LRULLOSEZ7PCQNA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.068557555 (516h41m24.93964059s) [noise=true , address=45.91.116.13:4155 , piece_id=KZFHAIDKB6KDY32VW5WHF2NN7XVHCSTQOBBOBEZFY5AIFAKQQCAA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.068861132 (516h41m24.939944167s) [noise=true , address=45.91.116.5:4300 , piece_id=QFSBFSOU7T3RSIULRY7XYFEMTU62HEKTBQHDYOWCP2INIHSK45LA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.069401712 (516h41m24.940484757s) [noise=true , address=77.90.31.19:4841 , piece_id=FBDPQXQBUOJOYGK3QMMCBJCHCDRCTWQWLN4M65TDSLGG4XJGV2DQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.069980293 (516h41m24.941063338s) [noise=true , address=145.79.227.49:4339 , piece_id=UHALUFULF47VFUOXJYAXZ44OVGXT3EWTTHYHKKGVAUBJ4SJS6QPQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.070261439 (516h41m24.941344474s) [noise=true , address=145.79.227.51:4409 , piece_id=AB2CJISPYOSZHE3RHHRAYOK2EFOUJVFY7TN4IGCEAQ7FSZU2ZQQQ, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:25.070806857 (516h41m24.941889902s) [noise=true , address=45.91.116.11:4117 , piece_id=NEWG6KKOJ7AFFMOMMGHDZG3NB4W5RTHV3EKMFANZMIN3QOZN7AZA, seg_pos={0 0}, upload=453350]: Uploading piece... Dec 27 04:10:26.784411779 (516h41m26.655494814s) [upload=453350]: Next returned nil segment Dec 27 04:10:26.784469059 (516h41m26.655552114s) [upload=453350]: Waiting for error group managing segments... Dec 27 04:10:27.066303036 (516h41m26.937386071s) [noise=true , address=145.79.227.42:4122 , piece_id=LCH33YDWNLGGRBSMTRQDIY7IQSGIYXCKK2K6JA2DSTQQJXRGREFQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.069488422 (516h41m26.940571467s) [noise=true , address=145.79.227.9:4366 , piece_id=B2Q4JV3CCJPVHH6DAR7QRD7GXKUMD4J6P7E244WNHYEGF3NCQIAQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.070565293 (516h41m26.941648338s) [noise=true , address=145.79.227.49:4339 , piece_id=UHALUFULF47VFUOXJYAXZ44OVGXT3EWTTHYHKKGVAUBJ4SJS6QPQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.072396479 (516h41m26.943479524s) [noise=true , address=145.79.227.37:4567 , piece_id=EP67HNPCONTAOOF6JMKQO36ZO2J34EDLTEGQ5GRNZBUFKDWB5N2Q, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.074158865 (516h41m26.9452419s) [noise=true , address=145.79.227.41:4086 , piece_id=DXKOQ7MW3NI6LHN5MB72R6LB6G77NIEIMVFR7LRULLOSEZ7PCQNA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.083607928 (516h41m26.954690964s) [noise=true , address=145.79.227.51:4409 , piece_id=AB2CJISPYOSZHE3RHHRAYOK2EFOUJVFY7TN4IGCEAQ7FSZU2ZQQQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.090935010 (516h41m26.962018045s) [noise=true , address=145.79.227.13:4802 , piece_id=EOXM35Q45TKYNVLLQBWJCFAMDVPNCA3CZ7MPUHFJOJO32HJWWHSQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.091165870 (516h41m26.962248915s) [noise=true , address=145.79.227.3:4151 , piece_id=BLC4CWLUHGINXQ6UO54NVKVKXHP36OFBNP2H6JZ3BCOY3XSFLEEA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.092753052 (516h41m26.963836087s) [noise=true , address=145.79.227.38:4979 , piece_id=V5GM6VCP5YJSTUPAX57PGDQBANHGDONBG44WJNRKBI2CSRJVFXSA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.108953400 (516h41m26.980036435s) [noise=true , address=145.79.227.15:4873 , piece_id=FIE76IFA4IMMGPVSMU4Z7B7TYLVUHWJMDAU5U5VBIO2WWWA7NBNQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.234519448 (516h41m27.105602483s) [noise=true , address=77.90.31.19:4841 , piece_id=FBDPQXQBUOJOYGK3QMMCBJCHCDRCTWQWLN4M65TDSLGG4XJGV2DQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.356736610 (516h41m27.227819655s) [noise=true , address=45.91.116.14:4190 , piece_id=43A2CNSKWJE25ADECKKNMOY7ENGZN2TPXCA5KLJERDCUBTF2BYDQ, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.385779121 (516h41m27.256862156s) [noise=true , address=45.91.116.11:4117 , piece_id=NEWG6KKOJ7AFFMOMMGHDZG3NB4W5RTHV3EKMFANZMIN3QOZN7AZA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: <nil> Dec 27 04:10:27.385830699 (516h41m27.256913744s) [seg_pos={0 0}, upload=453350]: Segment reached optimal threshold of 13 pieces. Dec 27 04:10:27.386012244 (516h41m27.25709528s) [noise=true , address=45.91.116.13:4155 , piece_id=KZFHAIDKB6KDY32VW5WHF2NN7XVHCSTQOBBOBEZFY5AIFAKQQCAA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:52138->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:52138->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52138->45.91.116.13:4155: use of closed network connection Dec 27 04:10:27.386019088 (516h41m27.257102123s) [noise=true , address=45.91.116.13:4155 , piece_id=KZFHAIDKB6KDY32VW5WHF2NN7XVHCSTQOBBOBEZFY5AIFAKQQCAA, seg_pos={0 0}, upload=453350]: Long tail cancellation due to optimal threshold Dec 27 04:10:27.386173772 (516h41m27.257256817s) [noise=true , address=45.91.116.5:4300 , piece_id=QFSBFSOU7T3RSIULRY7XYFEMTU62HEKTBQHDYOWCP2INIHSK45LA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 04:10:27.386190755 (516h41m27.25727379s) [noise=true , address=45.91.116.5:4300 , piece_id=QFSBFSOU7T3RSIULRY7XYFEMTU62HEKTBQHDYOWCP2INIHSK45LA, seg_pos={0 0}, upload=453350]: Long tail cancellation due to optimal threshold Dec 27 04:10:27.386454196 (516h41m27.257537231s) [noise=true , address=145.79.227.34:4905 , piece_id=7RHZ3MMIHPSNQMJENCX46ZOBGBQNAXHIY4YSIEMW56ZHBBFSIBEA, seg_pos={0 0}, upload=453350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:10:27.386462732 (516h41m27.257545767s) [noise=true , address=145.79.227.34:4905 , piece_id=7RHZ3MMIHPSNQMJENCX46ZOBGBQNAXHIY4YSIEMW56ZHBBFSIBEA, seg_pos={0 0}, upload=453350]: Long tail cancellation due to optimal threshold Dec 27 04:10:27.386487229 (516h41m27.257570264s) [seg_pos={0 0}, upload=453350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:10:27.386566830 (516h41m27.257649866s) [upload=453350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:10:27.386573724 (516h41m27.257656759s) [upload=453350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:10:27.686043263 (516h41m27.557126308s) [upload=453350]: Done uploading segments... Dec 27 04:10:27.686068260 (516h41m27.557151305s) [upload=453350]: Upload finished. err: <nil> Duration: 30.003322785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X.] duration: 6.09172691s trace: https://jaeger.dev.storj.io/trace/1bd4ca5318ac80bb FAIL (2025-12-27 04:10:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A) context deadline exceeded ======LOGS====== (516h40m6.050446384s) [upload=453329]: Uploading segments... Dec 27 04:09:06.179423113 (516h40m6.050506168s) [upload=453329]: Got next segment. Inline: false Dec 27 04:09:06.179464381 (516h40m6.050547426s) [upload=453329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:09:06.179470292 (516h40m6.050553337s) [upload=453329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:09:06.428450234 (516h40m6.299533279s) [seg_pos={0 0}, upload=453329]: Begin upload segment... Dec 27 04:09:06.428519275 (516h40m6.29960231s) [seg_pos={0 0}, upload=453329]: Done begin upload segment. Dec 27 04:09:06.428572055 (516h40m6.2996551s) [noise=true , address=145.79.227.7:4293 , piece_id=EYPR54ILDSUMQA2WMHT3LHLY7UALH57VA6HNEG2G2PMJ52DNJDAA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.428642890 (516h40m6.299725925s) [noise=true , address=145.79.227.31:4799 , piece_id=5OA47WIRGKTINN4OXK5IHJ37DVI27CL4XQPX2EG3MGT23KZBUHDQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429132934 (516h40m6.300215979s) [noise=true , address=145.79.227.48:4302 , piece_id=KUR6NEAJYK3N2SJFN4SORNZJXFPADPMNYF3FHTMC3RKDZJC2BFTQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429505513 (516h40m6.300588558s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429946122 (516h40m6.301029167s) [noise=true , address=145.79.227.35:4482 , piece_id=KUDH5DVZ2GU5SVCDS7SNUGGUCER5WBA7RPGYCROZRGVQU65XFV7Q, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.430342747 (516h40m6.301425792s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.430942590 (516h40m6.302025635s) [noise=true , address=145.79.227.29:4941 , piece_id=2PZA6LDZRLJTYB2OKN6VV4EGZZ5LZHT7BNUTNABMVVLGO5XNQYSQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.431360406 (516h40m6.302443451s) [noise=true , address=145.79.227.42:4123 , piece_id=74RJPAEOMTI32FBDX7UTQGJZBB3MZ6UWF2DGKDR6TWBSWI37EMWQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.432739241 (516h40m6.303822276s) [noise=true , address=145.79.227.6:4259 , piece_id=57WJMNZJKOGGPJQEFF7FO3F4RZXQ3FGRUVVY3VWRYJPHSY523KKQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.432828952 (516h40m6.303911997s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.434389093 (516h40m6.305472138s) [noise=true , address=45.91.116.10:4081 , piece_id=KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.435818665 (516h40m6.30690171s) [noise=true , address=77.90.31.21:4878 , piece_id=725GKJBHGVSRDNJEHMEJYEGVRDHO6RFMRHF3NH4RCF7VK5LOIWMQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.437261191 (516h40m6.308344236s) [noise=true , address=45.91.116.5:4299 , piece_id=DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.438734647 (516h40m6.309817692s) [noise=true , address=145.79.227.3:4151 , piece_id=QDA2T65E2NDENZNSEET57CQQ5YKMKMVUPWMJD7BWRCRZVAIPLMWA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.440154831 (516h40m6.311237876s) [noise=true , address=145.79.227.16:4912 , piece_id=PBJQP3QXFULBZM4GO4BPEYJOWCKHTFWDR364MGJWDG6UIGZ2FO4Q, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.440514506 (516h40m6.311597541s) [noise=true , address=145.79.227.41:4087 , piece_id=3U7AWR2UIZX5ZS557KDTOJNO6H7WT7RH5MM3IORTLKYJXMAC5SQA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:08.052717462 (516h40m7.923800497s) [upload=453329]: Next returned nil segment Dec 27 04:09:08.052731839 (516h40m7.923814874s) [upload=453329]: Waiting for error group managing segments... Dec 27 04:09:08.436600298 (516h40m8.307683333s) [noise=true , address=145.79.227.29:4941 , piece_id=2PZA6LDZRLJTYB2OKN6VV4EGZZ5LZHT7BNUTNABMVVLGO5XNQYSQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.446255743 (516h40m8.317338778s) [noise=true , address=145.79.227.41:4087 , piece_id=3U7AWR2UIZX5ZS557KDTOJNO6H7WT7RH5MM3IORTLKYJXMAC5SQA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450149267 (516h40m8.321232312s) [noise=true , address=145.79.227.16:4912 , piece_id=PBJQP3QXFULBZM4GO4BPEYJOWCKHTFWDR364MGJWDG6UIGZ2FO4Q, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450817609 (516h40m8.321900644s) [noise=true , address=145.79.227.3:4151 , piece_id=QDA2T65E2NDENZNSEET57CQQ5YKMKMVUPWMJD7BWRCRZVAIPLMWA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450840353 (516h40m8.321923398s) [noise=true , address=145.79.227.42:4123 , piece_id=74RJPAEOMTI32FBDX7UTQGJZBB3MZ6UWF2DGKDR6TWBSWI37EMWQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.454869985 (516h40m8.32595302s) [noise=true , address=145.79.227.35:4482 , piece_id=KUDH5DVZ2GU5SVCDS7SNUGGUCER5WBA7RPGYCROZRGVQU65XFV7Q, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.455525023 (516h40m8.326608058s) [noise=true , address=145.79.227.7:4293 , piece_id=EYPR54ILDSUMQA2WMHT3LHLY7UALH57VA6HNEG2G2PMJ52DNJDAA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.464687670 (516h40m8.335770705s) [noise=true , address=145.79.227.48:4302 , piece_id=KUR6NEAJYK3N2SJFN4SORNZJXFPADPMNYF3FHTMC3RKDZJC2BFTQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.469271008 (516h40m8.340354053s) [noise=true , address=145.79.227.6:4259 , piece_id=57WJMNZJKOGGPJQEFF7FO3F4RZXQ3FGRUVVY3VWRYJPHSY523KKQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.477265611 (516h40m8.348348666s) [noise=true , address=145.79.227.31:4799 , piece_id=5OA47WIRGKTINN4OXK5IHJ37DVI27CL4XQPX2EG3MGT23KZBUHDQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.693619574 (516h40m8.564702619s) [noise=true , address=77.90.31.21:4878 , piece_id=725GKJBHGVSRDNJEHMEJYEGVRDHO6RFMRHF3NH4RCF7VK5LOIWMQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.722812993 (516h40m8.593896038s) [noise=true , address=45.91.116.5:4299 , piece_id=DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.809452479 (516h40m8.680535544s) [noise=true , address=45.91.116.10:4081 , piece_id=KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.809478418 (516h40m8.680561473s) [seg_pos={0 0}, upload=453329]: Segment reached optimal threshold of 13 pieces. Dec 27 04:09:08.809725929 (516h40m8.680808964s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection Dec 27 04:09:08.809734355 (516h40m8.68081739s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.809762359 (516h40m8.680845404s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:09:08.809772949 (516h40m8.680856124s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.810094942 (516h40m8.681177988s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection Dec 27 04:09:08.810105973 (516h40m8.681189018s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.810153343 (516h40m8.681236398s) [seg_pos={0 0}, upload=453329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:09:08.810332043 (516h40m8.681415098s) [upload=453329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:09:08.810340290 (516h40m8.681423335s) [upload=453329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:09:09.110279803 (516h40m8.981362848s) [upload=453329]: Done uploading segments... Dec 27 04:09:09.110298067 (516h40m8.981381112s) [upload=453329]: Upload finished. err: <nil> Duration: 6.091781572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.....X............X................] duration: 21.724482749s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:09:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 21.724496465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........X] duration: 30.002841203s trace: https://jaeger.dev.storj.io/trace/1bd4ca5318ac80bb FAIL (2025-12-27 04:09:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A) context deadline exceeded ======LOGS====== (516h40m6.050446384s) [upload=453329]: Uploading segments... Dec 27 04:09:06.179423113 (516h40m6.050506168s) [upload=453329]: Got next segment. Inline: false Dec 27 04:09:06.179464381 (516h40m6.050547426s) [upload=453329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:09:06.179470292 (516h40m6.050553337s) [upload=453329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:09:06.428450234 (516h40m6.299533279s) [seg_pos={0 0}, upload=453329]: Begin upload segment... Dec 27 04:09:06.428519275 (516h40m6.29960231s) [seg_pos={0 0}, upload=453329]: Done begin upload segment. Dec 27 04:09:06.428572055 (516h40m6.2996551s) [noise=true , address=145.79.227.7:4293 , piece_id=EYPR54ILDSUMQA2WMHT3LHLY7UALH57VA6HNEG2G2PMJ52DNJDAA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.428642890 (516h40m6.299725925s) [noise=true , address=145.79.227.31:4799 , piece_id=5OA47WIRGKTINN4OXK5IHJ37DVI27CL4XQPX2EG3MGT23KZBUHDQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429132934 (516h40m6.300215979s) [noise=true , address=145.79.227.48:4302 , piece_id=KUR6NEAJYK3N2SJFN4SORNZJXFPADPMNYF3FHTMC3RKDZJC2BFTQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429505513 (516h40m6.300588558s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.429946122 (516h40m6.301029167s) [noise=true , address=145.79.227.35:4482 , piece_id=KUDH5DVZ2GU5SVCDS7SNUGGUCER5WBA7RPGYCROZRGVQU65XFV7Q, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.430342747 (516h40m6.301425792s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.430942590 (516h40m6.302025635s) [noise=true , address=145.79.227.29:4941 , piece_id=2PZA6LDZRLJTYB2OKN6VV4EGZZ5LZHT7BNUTNABMVVLGO5XNQYSQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.431360406 (516h40m6.302443451s) [noise=true , address=145.79.227.42:4123 , piece_id=74RJPAEOMTI32FBDX7UTQGJZBB3MZ6UWF2DGKDR6TWBSWI37EMWQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.432739241 (516h40m6.303822276s) [noise=true , address=145.79.227.6:4259 , piece_id=57WJMNZJKOGGPJQEFF7FO3F4RZXQ3FGRUVVY3VWRYJPHSY523KKQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.432828952 (516h40m6.303911997s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.434389093 (516h40m6.305472138s) [noise=true , address=45.91.116.10:4081 , piece_id=KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.435818665 (516h40m6.30690171s) [noise=true , address=77.90.31.21:4878 , piece_id=725GKJBHGVSRDNJEHMEJYEGVRDHO6RFMRHF3NH4RCF7VK5LOIWMQ, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.437261191 (516h40m6.308344236s) [noise=true , address=45.91.116.5:4299 , piece_id=DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.438734647 (516h40m6.309817692s) [noise=true , address=145.79.227.3:4151 , piece_id=QDA2T65E2NDENZNSEET57CQQ5YKMKMVUPWMJD7BWRCRZVAIPLMWA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.440154831 (516h40m6.311237876s) [noise=true , address=145.79.227.16:4912 , piece_id=PBJQP3QXFULBZM4GO4BPEYJOWCKHTFWDR364MGJWDG6UIGZ2FO4Q, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:06.440514506 (516h40m6.311597541s) [noise=true , address=145.79.227.41:4087 , piece_id=3U7AWR2UIZX5ZS557KDTOJNO6H7WT7RH5MM3IORTLKYJXMAC5SQA, seg_pos={0 0}, upload=453329]: Uploading piece... Dec 27 04:09:08.052717462 (516h40m7.923800497s) [upload=453329]: Next returned nil segment Dec 27 04:09:08.052731839 (516h40m7.923814874s) [upload=453329]: Waiting for error group managing segments... Dec 27 04:09:08.436600298 (516h40m8.307683333s) [noise=true , address=145.79.227.29:4941 , piece_id=2PZA6LDZRLJTYB2OKN6VV4EGZZ5LZHT7BNUTNABMVVLGO5XNQYSQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.446255743 (516h40m8.317338778s) [noise=true , address=145.79.227.41:4087 , piece_id=3U7AWR2UIZX5ZS557KDTOJNO6H7WT7RH5MM3IORTLKYJXMAC5SQA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450149267 (516h40m8.321232312s) [noise=true , address=145.79.227.16:4912 , piece_id=PBJQP3QXFULBZM4GO4BPEYJOWCKHTFWDR364MGJWDG6UIGZ2FO4Q, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450817609 (516h40m8.321900644s) [noise=true , address=145.79.227.3:4151 , piece_id=QDA2T65E2NDENZNSEET57CQQ5YKMKMVUPWMJD7BWRCRZVAIPLMWA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.450840353 (516h40m8.321923398s) [noise=true , address=145.79.227.42:4123 , piece_id=74RJPAEOMTI32FBDX7UTQGJZBB3MZ6UWF2DGKDR6TWBSWI37EMWQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.454869985 (516h40m8.32595302s) [noise=true , address=145.79.227.35:4482 , piece_id=KUDH5DVZ2GU5SVCDS7SNUGGUCER5WBA7RPGYCROZRGVQU65XFV7Q, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.455525023 (516h40m8.326608058s) [noise=true , address=145.79.227.7:4293 , piece_id=EYPR54ILDSUMQA2WMHT3LHLY7UALH57VA6HNEG2G2PMJ52DNJDAA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.464687670 (516h40m8.335770705s) [noise=true , address=145.79.227.48:4302 , piece_id=KUR6NEAJYK3N2SJFN4SORNZJXFPADPMNYF3FHTMC3RKDZJC2BFTQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.469271008 (516h40m8.340354053s) [noise=true , address=145.79.227.6:4259 , piece_id=57WJMNZJKOGGPJQEFF7FO3F4RZXQ3FGRUVVY3VWRYJPHSY523KKQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.477265611 (516h40m8.348348666s) [noise=true , address=145.79.227.31:4799 , piece_id=5OA47WIRGKTINN4OXK5IHJ37DVI27CL4XQPX2EG3MGT23KZBUHDQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.693619574 (516h40m8.564702619s) [noise=true , address=77.90.31.21:4878 , piece_id=725GKJBHGVSRDNJEHMEJYEGVRDHO6RFMRHF3NH4RCF7VK5LOIWMQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.722812993 (516h40m8.593896038s) [noise=true , address=45.91.116.5:4299 , piece_id=DJMMUQ4TBVGSSOMHH37BZANEQ76VH5QJTKBOHYZGC5FO3RFYHH6A, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.809452479 (516h40m8.680535544s) [noise=true , address=45.91.116.10:4081 , piece_id=KNZMNZDHXHJQR7VMARAWT7NDQHNA5DXX4ONC2I3NQX6DFHYQ6UCQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: <nil> Dec 27 04:09:08.809478418 (516h40m8.680561473s) [seg_pos={0 0}, upload=453329]: Segment reached optimal threshold of 13 pieces. Dec 27 04:09:08.809725929 (516h40m8.680808964s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56100->77.90.31.14:4663: use of closed network connection Dec 27 04:09:08.809734355 (516h40m8.68081739s) [noise=true , address=77.90.31.14:4663 , piece_id=BFZTLK3KXEL6OE2MN6QUM6KNOTXTI6SKVA7YQXULYMN66DLX4RDA, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.809762359 (516h40m8.680845404s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:09:08.809772949 (516h40m8.680856124s) [noise=true , address=145.79.227.40:4050 , piece_id=CP75ONFN7WT4QLPXNZ3W2DWGKCB4DZPJWEHKUTU3GQ4C744AB5XQ, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.810094942 (516h40m8.681177988s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54720->77.90.31.10:4519: use of closed network connection Dec 27 04:09:08.810105973 (516h40m8.681189018s) [noise=true , address=77.90.31.10:4519 , piece_id=OMPQ65XW6AAVFMOW56XFIBYL6NXDE4ZRAX3LJPVFTE457LBTBA3A, seg_pos={0 0}, upload=453329]: Long tail cancellation due to optimal threshold Dec 27 04:09:08.810153343 (516h40m8.681236398s) [seg_pos={0 0}, upload=453329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:09:08.810332043 (516h40m8.681415098s) [upload=453329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:09:08.810340290 (516h40m8.681423335s) [upload=453329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:09:09.110279803 (516h40m8.981362848s) [upload=453329]: Done uploading segments... Dec 27 04:09:09.110298067 (516h40m8.981381112s) [upload=453329]: Upload finished. err: <nil> Duration: 30.002868294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.....X............X...............] duration: 55.945070535s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:09:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 55.945087006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X........] duration: 6.632723074s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:08:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 6.632754405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.....X............X..............] duration: 31.374164327s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 31.374196549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.....X............X.............] duration: 15.849623997s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:07:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 15.84965191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X.......] duration: 6.016069439s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 6.016106511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X......] duration: 6.009707805s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 6.009738644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....X............X............] duration: 21.208312574s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:06:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 21.208336109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X.....] duration: 6.151236541s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:06:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 6.151273672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....X............X...........] duration: 27.099358247s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:06:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 27.09938627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X....] duration: 5.895297991s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 5.89531878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.....X............X..........] duration: 38.386628638s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:05:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 38.386653184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X...] duration: 12.396324743s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 12.396351123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X..] duration: 10.369689107s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 10.369716159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.....X............X.........] duration: 43.790644628s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:04:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 43.790670717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X.] duration: 12.095088058s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 12.095115539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.XXX.XXX.X..XXXXX..X..X] duration: 30.003025948s trace: https://jaeger.dev.storj.io/trace/20d4264dbbe153a9 FAIL (2025-12-27 04:03:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ) context deadline exceeded ======LOGS====== Dec 27 04:02:48.709718508 (516h33m48.580801543s) [upload=453238]: Starting upload Dec 27 04:02:48.709789894 (516h33m48.580872929s) [upload=453238]: Done starting upload Dec 27 04:02:48.710014541 (516h33m48.581097587s) [upload=453238]: Uploading segments... Dec 27 04:02:48.710048005 (516h33m48.58113105s) [upload=453238]: Got next segment. Inline: false Dec 27 04:02:48.710084455 (516h33m48.58116749s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:02:48.710088853 (516h33m48.581171888s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:02:48.957351784 (516h33m48.828434819s) [seg_pos={0 0}, upload=453238]: Begin upload segment... Dec 27 04:02:48.957397953 (516h33m48.828480998s) [seg_pos={0 0}, upload=453238]: Done begin upload segment. Dec 27 04:02:48.957423781 (516h33m48.828506816s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.957552857 (516h33m48.828635892s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958092585 (516h33m48.82917562s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.958612464 (516h33m48.829695509s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959024259 (516h33m48.830107304s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959471220 (516h33m48.830554265s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.959845002 (516h33m48.830928047s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960212331 (516h33m48.831295366s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.960585612 (516h33m48.831668647s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.961644619 (516h33m48.832727664s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962056993 (516h33m48.833140039s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962504676 (516h33m48.833587711s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.962906101 (516h33m48.833989136s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963303768 (516h33m48.834386803s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.963728136 (516h33m48.834811171s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:48.964115755 (516h33m48.83519879s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Uploading piece... Dec 27 04:02:50.686725530 (516h33m50.557808565s) [upload=453238]: Next returned nil segment Dec 27 04:02:50.686742672 (516h33m50.557825717s) [upload=453238]: Waiting for error group managing segments... Dec 27 04:02:50.973066542 (516h33m50.844149577s) [noise=true , address=145.79.227.51:4411 , piece_id=FT5QN3PI3JM7KEHQ22FEELONAM3LIBTEJIGNXTTD6YV2XR3DOEFQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.979019067 (516h33m50.850102112s) [noise=true , address=145.79.227.45:4230 , piece_id=BR3N5DAKI4ZHUZQMRSQKTVJOBRJ5FURBSILLJM377TXQD5BFBN3Q, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.980998085 (516h33m50.85208112s) [noise=true , address=145.79.227.8:4331 , piece_id=IMRXARC6APGDWL7QGVGDB65WB42I2XUVYGHU2G54YMWOQOWEDECA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.984546612 (516h33m50.855629647s) [noise=true , address=145.79.227.13:4802 , piece_id=VRMHTEN5EZJRN6WUE32YXYGIRYZEFPZ2S6GEKMTZNCTLVYN4LZSA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987774839 (516h33m50.858857874s) [noise=true , address=145.79.227.42:4122 , piece_id=INL5DZKJ7ONLIIAJAL6EH7EZVNXKZQ52RUNG37UWIC5NN3QN5SEQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.987865321 (516h33m50.858948366s) [noise=true , address=145.79.227.31:4799 , piece_id=RY4QRVHMGLKSJNXDYZ5FXI5KILEWZOY6OQ3CDTJLNUDEP6YWHZWQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:50.996200614 (516h33m50.867283649s) [noise=true , address=145.79.227.48:4301 , piece_id=6GCSZ7BIV2G3L2EQEBBXPUWJOYPI4P4PNUXVOU6XCQP232GSCCLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.000251757 (516h33m50.871334823s) [noise=true , address=145.79.227.2:4113 , piece_id=4VHVLMQW2FF63LILG2BC7QNONRENQOAKFTXHL4IA7ALTYM2INRAA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.008621815 (516h33m50.87970486s) [noise=true , address=145.79.227.29:4942 , piece_id=WCGOBGPS64F4UVNKNHRKPDALHOWDLEHSM45QTZVTK2OZK3IUQFKQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.212156360 (516h33m51.083239395s) [noise=true , address=45.91.116.7:4010 , piece_id=Y5KENAIZDBGAHRTONB2ZPYPGC6AH7C6UPBTXC6YPLPETZJ4B5FGA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.272450320 (516h33m51.143533355s) [noise=true , address=77.90.31.17:4773 , piece_id=XJ4WFBNM743WR4JWL5RO67TXRSUTENT537QAXCYY75VE7FNVFVTA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.313334560 (516h33m51.184417595s) [noise=true , address=45.91.116.6:4975 , piece_id=GZ3T3KLM26TVDFN45CI5EF6MHCTSJRJSYNJAMKQOLDESEU3RTVXQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328772437 (516h33m51.199855472s) [noise=true , address=45.91.116.10:4081 , piece_id=VEUBYDWGYSCF42CZ7AFK45GFGXR6NBTMPLWMXLHAXQJGXDQNB5AA, seg_pos={0 0}, upload=453238]: Done uploading piece. err: <nil> Dec 27 04:02:51.328792214 (516h33m51.199875249s) [seg_pos={0 0}, upload=453238]: Segment reached optimal threshold of 13 pieces. Dec 27 04:02:51.329023836 (516h33m51.200106871s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329030598 (516h33m51.200113633s) [noise=true , address=45.91.116.8:4045 , piece_id=PAIUCYFG66HCX6F7O5FDK3ERSOS3DM4K6NPRI2A2KTKUNNTEJSLQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329174532 (516h33m51.200257567s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329180885 (516h33m51.20026392s) [noise=true , address=145.79.227.34:4907 , piece_id=D5HFHWJUOKVQXMPMZYNFJXLBEDPYOFPEXBLKXG6ZJ2WTYLC6VQ7A, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329624459 (516h33m51.200707494s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 04:02:51.329635411 (516h33m51.200718456s) [noise=true , address=145.79.227.33:4869 , piece_id=FEZ6KYS333NBTOCNXOWAOMDCYZFHA2V6LUCSITYXDFOSKKLQH3ZQ, seg_pos={0 0}, upload=453238]: Long tail cancellation due to optimal threshold Dec 27 04:02:51.329666340 (516h33m51.200749375s) [seg_pos={0 0}, upload=453238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:02:51.329766329 (516h33m51.200849374s) [upload=453238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:02:51.329772933 (516h33m51.200855978s) [upload=453238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:02:51.627353477 (516h33m51.498436522s) [upload=453238]: Done uploading segments... Dec 27 04:02:51.627379897 (516h33m51.498462942s) [upload=453238]: Upload finished. err: <nil> Duration: 30.003048611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.....X............X........] duration: 22.946414945s trace: https://jaeger.dev.storj.io/trace/2f60723659e3454f FAIL (2025-12-27 04:03:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, Piece ID: ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ) context deadline exceeded; piecestore: (Node ID: 13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, Piece ID: HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ) context deadline exceeded; piecestore: (Node ID: 12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, Piece ID: FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ) context deadline exceeded ======LOGS====== oise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751258634 (516h26m44.622341669s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.751664597 (516h26m44.622747642s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752049530 (516h26m44.623132575s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.752689839 (516h26m44.623772874s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753261037 (516h26m44.624344082s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.753673983 (516h26m44.624757018s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754123559 (516h26m44.625206594s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.754623612 (516h26m44.625706657s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755115779 (516h26m44.626198824s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.755571296 (516h26m44.626654331s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756073091 (516h26m44.627156126s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.756606748 (516h26m44.627689793s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757107912 (516h26m44.628190957s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.757606421 (516h26m44.628689456s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:44.763605595 (516h26m44.63468863s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Uploading piece... Dec 27 03:55:46.114820489 (516h26m45.985903534s) [upload=453142]: Next returned nil segment Dec 27 03:55:46.114838174 (516h26m45.985921219s) [upload=453142]: Waiting for error group managing segments... Dec 27 03:55:46.456602171 (516h26m46.327685217s) [noise=true , address=66.23.193.175:4561 , piece_id=I3EYT2ADEH65BQ3LKQ5AUD7SM52N45GZENTEFQAH72TQ4OHQJJDQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.776348462 (516h26m46.647431497s) [noise=true , address=145.79.227.43:4189 , piece_id=HJS5W6DHFD4BSJNR7RJT7O6WWINZGXTTUDJ5NPWVJUGESJ7HKCFQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.852227690 (516h26m46.723310735s) [noise=true , address=45.91.116.5:4321 , piece_id=SNSJYD6235NDAVDDSGAHTKU6BWCTD5FHBQOTRNKXLRVXB5KDVJEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.905697706 (516h26m46.776780741s) [noise=true , address=77.90.31.21:4903 , piece_id=VXMRODD52XJRJ3VTTL7JFEJ4UQOZKNC2CGQJP5H2WTIQ5JUR3ROQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.910247900 (516h26m46.781330936s) [noise=true , address=45.91.116.4:4944 , piece_id=A2VGPZMDBQ5LIOVYFMQZJGNKIEV2AE7265GDX5HAFEBDC3BDAZWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.919792535 (516h26m46.79087557s) [noise=true , address=77.90.31.13:4636 , piece_id=6TL36QQC2BK5X6VCONL7SREXFJVDNKLXCHE4YZX42DHP3DDI52EQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.925895746 (516h26m46.796978791s) [noise=true , address=45.91.116.4:4948 , piece_id=SKQC4I7MXM6CZQ5BGG5SAWR2UBSNGXN2L62FMRFILM5TS46NWS3Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:46.941922244 (516h26m46.813005279s) [noise=true , address=77.90.31.10:4537 , piece_id=COZCE3H4WS72CXY4MTYTZ5PSWOQN6H24E4LMFIBBGYIG7C3PRPBQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.132909747 (516h26m47.003992792s) [noise=true , address=145.79.227.42:4129 , piece_id=ZHCJQ2KSEUUFWXHMFSNBWSFJY7GPSMQRDZ2CRAVU7YJBPUM2JDNQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.612235966 (516h26m47.483319011s) [noise=true , address=145.79.227.38:4008 , piece_id=CLX3YLBH2MYNHRL52QHJLY4XVQCW7LXARUTY3ZRFF3AYLWAPGRTQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.674643671 (516h26m47.545726716s) [noise=true , address=145.79.227.35:4489 , piece_id=FRQIMMV7EK2L472T3BMGK7UX6L33ZGG3TIF4C3U3HTNIDQYKNXEQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.683394995 (516h26m47.55447803s) [noise=true , address=77.90.31.12:4613 , piece_id=YR2T6LYKKUBPACYTGFWGVOX3OR5KJ4KXADCNTQZ4VED6B6QY5D5Q, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.845964575 (516h26m47.717047621s) [noise=true , address=145.79.227.40:4058 , piece_id=XI6HCWLVCOJGGSM7ZRDURPIFWR3TRMLPVASPK6GWLKNQSZAJP6GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906552776 (516h26m47.777635811s) [noise=true , address=45.91.116.9:4268 , piece_id=BM4DCRDTGXUNSNR3JQ4SFDRIENYDBR4COTUSPU5BZYSWFXY3JVZA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: <nil> Dec 27 03:55:47.906576230 (516h26m47.777659265s) [seg_pos={0 0}, upload=453142]: Segment reached optimal threshold of 14 pieces. Dec 27 03:55:47.906759109 (516h26m47.777842144s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906765541 (516h26m47.777848586s) [noise=true , address=145.79.227.33:4875 , piece_id=R26RC3YFDRISMLG6HYOFMI4NF7MZFUL7E6VTESSAMZBTLMMYSKPA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906820786 (516h26m47.777903851s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52700->64.49.13.169:4709: use of closed network connection Dec 27 03:55:47.906833310 (516h26m47.777916356s) [noise=true , address=64.49.13.169:4709 , piece_id=NU5PIHAAWGJUKNVWTMWFWTJVGICRDKM4YCEWLKAFGVKV2JBYU2IQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906868317 (516h26m47.777951352s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906873176 (516h26m47.777956211s) [noise=true , address=145.79.227.44:4197 , piece_id=JNE7C6CZ3T6Z2E45WRAJXDAUV33IC24OO7EQRQYAJXIU6GBFM7GA, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906964370 (516h26m47.778047405s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM): protocol: expected piece hash; context canceled; EOF Dec 27 03:55:47.906969770 (516h26m47.778052805s) [noise=true , address=145.79.227.5:4250 , piece_id=YCUJJKHNG4T62GLAK4GXNBNNUDYIKWS5MVNJYGNGKCGNOWAQBPWQ, seg_pos={0 0}, upload=453142]: Long tail cancellation due to optimal threshold Dec 27 03:55:47.906999567 (516h26m47.778082602s) [seg_pos={0 0}, upload=453142]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:55:47.907065382 (516h26m47.778148437s) [upload=453142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:55:47.907074670 (516h26m47.778157735s) [upload=453142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:55:48.204795712 (516h26m48.075878747s) [upload=453142]: Done uploading segments... Dec 27 03:55:48.204814278 (516h26m48.075897313s) [upload=453142]: Upload finished. err: <nil> Duration: 22.946435143s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)