Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001599483s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:05:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001622776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 38.933577463s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:05:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 38.933602369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...] duration: 6.952426862s trace: https://jaeger.dev.storj.io/trace/1cfa3f4f8fd2264f FAIL (2025-12-28 02:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q) context deadline exceeded ======LOGS====== 538h33m16.183793868s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:02:16.561525949 (538h33m16.432608994s) [seg_pos={0 0}, upload=472308]: Begin upload segment... Dec 28 02:02:16.561585833 (538h33m16.432668879s) [seg_pos={0 0}, upload=472308]: Done begin upload segment. Dec 28 02:02:16.561607455 (538h33m16.43269049s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.561622473 (538h33m16.432705518s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562142624 (538h33m16.433225669s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562525543 (538h33m16.433608588s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563010196 (538h33m16.434093241s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563417912 (538h33m16.434500947s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563825237 (538h33m16.434908282s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.564230539 (538h33m16.435313585s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.565700388 (538h33m16.436783423s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.567234289 (538h33m16.438317324s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.568973831 (538h33m16.440056876s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.570553209 (538h33m16.441636254s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572093913 (538h33m16.443176958s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572415886 (538h33m16.443498921s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.573634396 (538h33m16.444717431s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.574821446 (538h33m16.445904481s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:18.196329680 (538h33m18.067412725s) [upload=472308]: Next returned nil segment Dec 28 02:02:18.196343396 (538h33m18.067426441s) [upload=472308]: Waiting for error group managing segments... Dec 28 02:02:18.577068480 (538h33m18.448151515s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577569544 (538h33m18.448652589s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577594211 (538h33m18.448677266s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577929921 (538h33m18.449012956s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.582149756 (538h33m18.453232801s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.583056472 (538h33m18.454139517s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.627568481 (538h33m18.498651526s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.641129994 (538h33m18.512213029s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.735582849 (538h33m18.606665884s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.757245958 (538h33m18.628328993s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.859341556 (538h33m18.730424591s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.874631259 (538h33m18.745714294s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947273942 (538h33m18.818356977s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947297357 (538h33m18.818380402s) [seg_pos={0 0}, upload=472308]: Segment reached optimal threshold of 13 pieces. Dec 28 02:02:18.947551060 (538h33m18.818634095s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection Dec 28 02:02:18.947560218 (538h33m18.818643253s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947824651 (538h33m18.818907696s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection Dec 28 02:02:18.947833267 (538h33m18.818916312s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947871801 (538h33m18.818954846s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection Dec 28 02:02:18.947890948 (538h33m18.818973993s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947922387 (538h33m18.819005433s) [seg_pos={0 0}, upload=472308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:02:18.948079376 (538h33m18.819162421s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:18.948086460 (538h33m18.819169495s) [upload=472308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:19.245696579 (538h33m19.116779644s) [upload=472308]: Done uploading segments... Dec 28 02:02:19.245784487 (538h33m19.116867522s) [upload=472308]: Upload finished. err: <nil> Duration:* 6.952449827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 29.660855335s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 29.660873108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X..] duration: 7.057738257s trace: https://jaeger.dev.storj.io/trace/1cfa3f4f8fd2264f FAIL (2025-12-28 02:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q) context deadline exceeded ======LOGS====== 538h33m16.183793868s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:02:16.561525949 (538h33m16.432608994s) [seg_pos={0 0}, upload=472308]: Begin upload segment... Dec 28 02:02:16.561585833 (538h33m16.432668879s) [seg_pos={0 0}, upload=472308]: Done begin upload segment. Dec 28 02:02:16.561607455 (538h33m16.43269049s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.561622473 (538h33m16.432705518s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562142624 (538h33m16.433225669s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562525543 (538h33m16.433608588s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563010196 (538h33m16.434093241s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563417912 (538h33m16.434500947s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563825237 (538h33m16.434908282s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.564230539 (538h33m16.435313585s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.565700388 (538h33m16.436783423s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.567234289 (538h33m16.438317324s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.568973831 (538h33m16.440056876s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.570553209 (538h33m16.441636254s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572093913 (538h33m16.443176958s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572415886 (538h33m16.443498921s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.573634396 (538h33m16.444717431s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.574821446 (538h33m16.445904481s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:18.196329680 (538h33m18.067412725s) [upload=472308]: Next returned nil segment Dec 28 02:02:18.196343396 (538h33m18.067426441s) [upload=472308]: Waiting for error group managing segments... Dec 28 02:02:18.577068480 (538h33m18.448151515s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577569544 (538h33m18.448652589s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577594211 (538h33m18.448677266s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577929921 (538h33m18.449012956s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.582149756 (538h33m18.453232801s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.583056472 (538h33m18.454139517s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.627568481 (538h33m18.498651526s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.641129994 (538h33m18.512213029s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.735582849 (538h33m18.606665884s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.757245958 (538h33m18.628328993s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.859341556 (538h33m18.730424591s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.874631259 (538h33m18.745714294s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947273942 (538h33m18.818356977s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947297357 (538h33m18.818380402s) [seg_pos={0 0}, upload=472308]: Segment reached optimal threshold of 13 pieces. Dec 28 02:02:18.947551060 (538h33m18.818634095s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection Dec 28 02:02:18.947560218 (538h33m18.818643253s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947824651 (538h33m18.818907696s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection Dec 28 02:02:18.947833267 (538h33m18.818916312s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947871801 (538h33m18.818954846s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection Dec 28 02:02:18.947890948 (538h33m18.818973993s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947922387 (538h33m18.819005433s) [seg_pos={0 0}, upload=472308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:02:18.948079376 (538h33m18.819162421s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:18.948086460 (538h33m18.819169495s) [upload=472308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:19.245696579 (538h33m19.116779644s) [upload=472308]: Done uploading segments... Dec 28 02:02:19.245784487 (538h33m19.116867522s) [upload=472308]: Upload finished. err: <nil> Duration:* 7.057791207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.00339062s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 1m0.003412691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X.] duration: 8.79329184s trace: https://jaeger.dev.storj.io/trace/1cfa3f4f8fd2264f FAIL (2025-12-28 02:02:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q) context deadline exceeded ======LOGS====== 538h33m16.183793868s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:02:16.561525949 (538h33m16.432608994s) [seg_pos={0 0}, upload=472308]: Begin upload segment... Dec 28 02:02:16.561585833 (538h33m16.432668879s) [seg_pos={0 0}, upload=472308]: Done begin upload segment. Dec 28 02:02:16.561607455 (538h33m16.43269049s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.561622473 (538h33m16.432705518s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562142624 (538h33m16.433225669s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562525543 (538h33m16.433608588s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563010196 (538h33m16.434093241s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563417912 (538h33m16.434500947s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563825237 (538h33m16.434908282s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.564230539 (538h33m16.435313585s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.565700388 (538h33m16.436783423s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.567234289 (538h33m16.438317324s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.568973831 (538h33m16.440056876s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.570553209 (538h33m16.441636254s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572093913 (538h33m16.443176958s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572415886 (538h33m16.443498921s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.573634396 (538h33m16.444717431s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.574821446 (538h33m16.445904481s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:18.196329680 (538h33m18.067412725s) [upload=472308]: Next returned nil segment Dec 28 02:02:18.196343396 (538h33m18.067426441s) [upload=472308]: Waiting for error group managing segments... Dec 28 02:02:18.577068480 (538h33m18.448151515s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577569544 (538h33m18.448652589s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577594211 (538h33m18.448677266s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577929921 (538h33m18.449012956s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.582149756 (538h33m18.453232801s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.583056472 (538h33m18.454139517s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.627568481 (538h33m18.498651526s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.641129994 (538h33m18.512213029s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.735582849 (538h33m18.606665884s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.757245958 (538h33m18.628328993s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.859341556 (538h33m18.730424591s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.874631259 (538h33m18.745714294s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947273942 (538h33m18.818356977s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947297357 (538h33m18.818380402s) [seg_pos={0 0}, upload=472308]: Segment reached optimal threshold of 13 pieces. Dec 28 02:02:18.947551060 (538h33m18.818634095s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection Dec 28 02:02:18.947560218 (538h33m18.818643253s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947824651 (538h33m18.818907696s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection Dec 28 02:02:18.947833267 (538h33m18.818916312s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947871801 (538h33m18.818954846s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection Dec 28 02:02:18.947890948 (538h33m18.818973993s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947922387 (538h33m18.819005433s) [seg_pos={0 0}, upload=472308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:02:18.948079376 (538h33m18.819162421s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:18.948086460 (538h33m18.819169495s) [upload=472308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:19.245696579 (538h33m19.116779644s) [upload=472308]: Done uploading segments... Dec 28 02:02:19.245784487 (538h33m19.116867522s) [upload=472308]: Upload finished. err: <nil> Duration:* 8.793308441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X] duration: 30.003419948s trace: https://jaeger.dev.storj.io/trace/1cfa3f4f8fd2264f FAIL (2025-12-28 02:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q) context deadline exceeded ======LOGS====== 538h33m16.183793868s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:02:16.561525949 (538h33m16.432608994s) [seg_pos={0 0}, upload=472308]: Begin upload segment... Dec 28 02:02:16.561585833 (538h33m16.432668879s) [seg_pos={0 0}, upload=472308]: Done begin upload segment. Dec 28 02:02:16.561607455 (538h33m16.43269049s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.561622473 (538h33m16.432705518s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562142624 (538h33m16.433225669s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.562525543 (538h33m16.433608588s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563010196 (538h33m16.434093241s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563417912 (538h33m16.434500947s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.563825237 (538h33m16.434908282s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.564230539 (538h33m16.435313585s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.565700388 (538h33m16.436783423s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.567234289 (538h33m16.438317324s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.568973831 (538h33m16.440056876s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.570553209 (538h33m16.441636254s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572093913 (538h33m16.443176958s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.572415886 (538h33m16.443498921s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.573634396 (538h33m16.444717431s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:16.574821446 (538h33m16.445904481s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Uploading piece... Dec 28 02:02:18.196329680 (538h33m18.067412725s) [upload=472308]: Next returned nil segment Dec 28 02:02:18.196343396 (538h33m18.067426441s) [upload=472308]: Waiting for error group managing segments... Dec 28 02:02:18.577068480 (538h33m18.448151515s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5FYQ7EF2WETOK2A4NWGRU4SF34KMCNMQKI6Q46Y7QJCN4BYZ5Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577569544 (538h33m18.448652589s) [noise=true , address=145.79.227.5:4221 , piece_id=HG4RPRLEPEDFG3KY2W2TVEHSHXSEHWKHKTR7SM7WH2GU6HKYZCSQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577594211 (538h33m18.448677266s) [noise=true , address=145.79.227.17:4970 , piece_id=K6JR7KBFOGQUR3HZWULFJA37QWO64WOKSVC7TJGYLQOXBAWR4AWA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.577929921 (538h33m18.449012956s) [noise=true , address=145.79.227.31:4798 , piece_id=DWAV6VFW37BZSZEPCLWWEJD7C7ETR2JEWB23VMXPR5FLWZZHJBBQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.582149756 (538h33m18.453232801s) [noise=true , address=145.79.227.14:4840 , piece_id=VBR5H23KZTVA2L2WRX5TJFJV6B3RF5ZY47EDM43KY2O34NICBJIA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.583056472 (538h33m18.454139517s) [noise=true , address=145.79.227.11:4729 , piece_id=M22JE7K7ODZPL2KM3Y3ILUA3KTVL55BVWHVU4YSEQ5MOZWQ52GKA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.627568481 (538h33m18.498651526s) [noise=true , address=145.79.227.50:4375 , piece_id=I7TVVISU5R7MTG7UDFKCAJW5MBDQQEH2LDC2PMPM7UDVNXDYY2VA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.641129994 (538h33m18.512213029s) [noise=true , address=145.79.227.4:4187 , piece_id=LURY7VYL22W7GUF4LZMYUFMVMHREYCCIH2TSUWO35UYV4RNGY4UQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.735582849 (538h33m18.606665884s) [noise=true , address=145.79.227.32:4836 , piece_id=TNY7UG374DDTFHG5RHXITTXNZC7W2OO4RNUGCEY7L2Q3KBDRA4FQ, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.757245958 (538h33m18.628328993s) [noise=true , address=145.79.227.24:4583 , piece_id=547ZJYPGHCBP7BTZUJNPXRPW75KQXAUZXYQDJCXK76542ZTENLSA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.859341556 (538h33m18.730424591s) [noise=true , address=45.91.116.6:4975 , piece_id=DOMX736ZV37TIROPKI6EFJ7GPCMGC7IQE3ZQ3TXE7QRSKPFRJUBA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.874631259 (538h33m18.745714294s) [noise=true , address=77.90.31.16:4736 , piece_id=G5OHKFM2BSNXNPL2GH3RG2SQVU6WN7IT6NNDM57RV6LPQLHKIGFA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947273942 (538h33m18.818356977s) [noise=true , address=45.91.116.9:4262 , piece_id=F55NY3LGI77ERR3ZWOJMA3N6MLE4RPC2JJGUU2MRVGSDT7NRN44Q, seg_pos={0 0}, upload=472308]: Done uploading piece. err: <nil> Dec 28 02:02:18.947297357 (538h33m18.818380402s) [seg_pos={0 0}, upload=472308]: Segment reached optimal threshold of 13 pieces. Dec 28 02:02:18.947551060 (538h33m18.818634095s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58510->145.79.227.6:4257: use of closed network connection Dec 28 02:02:18.947560218 (538h33m18.818643253s) [noise=true , address=145.79.227.6:4257 , piece_id=RWC2OG2GVNRJIXIDKGX6C2LP32THCQPNB4H5H7DE4GFFHC4GTPYA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947824651 (538h33m18.818907696s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34260->77.90.31.10:4519: use of closed network connection Dec 28 02:02:18.947833267 (538h33m18.818916312s) [noise=true , address=77.90.31.10:4519 , piece_id=PDMTKHO35VBQ6DETIW5HCW6E7QH5EUEGPEQ5WAU5LQKNTM2UKO7A, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947871801 (538h33m18.818954846s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59144->145.79.227.19:4077: use of closed network connection Dec 28 02:02:18.947890948 (538h33m18.818973993s) [noise=true , address=145.79.227.19:4077 , piece_id=QPW3K2CTSVEBTDQSYNOYME5DM7WCHCIM3QCMWCPCGP4LDETDYDPA, seg_pos={0 0}, upload=472308]: Long tail cancellation due to optimal threshold Dec 28 02:02:18.947922387 (538h33m18.819005433s) [seg_pos={0 0}, upload=472308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:02:18.948079376 (538h33m18.819162421s) [upload=472308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:18.948086460 (538h33m18.819169495s) [upload=472308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:19.245696579 (538h33m19.116779644s) [upload=472308]: Done uploading segments... Dec 28 02:02:19.245784487 (538h33m19.116867522s) [upload=472308]: Upload finished. err: <nil> Duration:* 30.003443744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..] duration: 6.149198002s trace: https://jaeger.dev.storj.io/trace/38e62ebd31305c3a FAIL (2025-12-28 02:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA) context deadline exceeded ======LOGS====== c 28 01:59:54.632149533 (538h30m54.503232578s) [upload=472271]: Starting upload Dec 28 01:59:54.632223104 (538h30m54.503306149s) [upload=472271]: Done starting upload Dec 28 01:59:54.632459984 (538h30m54.50354303s) [upload=472271]: Uploading segments... Dec 28 01:59:54.632518817 (538h30m54.503601862s) [upload=472271]: Got next segment. Inline: false Dec 28 01:59:54.632550427 (538h30m54.503633462s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:59:54.632555987 (538h30m54.503639033s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:59:54.877485034 (538h30m54.748568069s) [seg_pos={0 0}, upload=472271]: Begin upload segment... Dec 28 01:59:54.877531954 (538h30m54.748614999s) [seg_pos={0 0}, upload=472271]: Done begin upload segment. Dec 28 01:59:54.877558324 (538h30m54.748641369s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.877632014 (538h30m54.748715059s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878258988 (538h30m54.749342033s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878542548 (538h30m54.749625583s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878774830 (538h30m54.749857865s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878917913 (538h30m54.750000958s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879363390 (538h30m54.750446425s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879398207 (538h30m54.750481242s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879728556 (538h30m54.750811591s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879962371 (538h30m54.751045406s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880052162 (538h30m54.751135197s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880298681 (538h30m54.751381717s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880517267 (538h30m54.751600302s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880627277 (538h30m54.751710322s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881125847 (538h30m54.752208892s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881996054 (538h30m54.753079089s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:56.553985411 (538h30m56.425068446s) [upload=472271]: Next returned nil segment Dec 28 01:59:56.554003425 (538h30m56.42508647s) [upload=472271]: Waiting for error group managing segments... Dec 28 01:59:56.879117778 (538h30m56.750200823s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.885385372 (538h30m56.756468407s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.890295382 (538h30m56.761378417s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.893360628 (538h30m56.764443663s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.896192852 (538h30m56.767275897s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.898752405 (538h30m56.76983544s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.900930874 (538h30m56.772013909s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.906041586 (538h30m56.777124621s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.908767286 (538h30m56.779850321s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.914066997 (538h30m56.785150032s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.160972856 (538h30m57.032055902s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.175758361 (538h30m57.046841396s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176365047 (538h30m57.047448082s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176385485 (538h30m57.04746852s) [seg_pos={0 0}, upload=472271]: Segment reached optimal threshold of 13 pieces. Dec 28 01:59:57.176565698 (538h30m57.047648733s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection Dec 28 01:59:57.176571319 (538h30m57.047654354s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176753717 (538h30m57.047836762s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176765208 (538h30m57.047848253s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176780998 (538h30m57.047864033s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176789875 (538h30m57.04787291s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176861181 (538h30m57.047944216s) [seg_pos={0 0}, upload=472271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:59:57.176948928 (538h30m57.048031963s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:59:57.176954178 (538h30m57.048037213s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:59:57.464405794 (538h30m57.335488839s) [upload=472271]: Done uploading segments... Dec 28 01:59:57.464430300 (538h30m57.335513336s) [upload=472271]: Upload finished. err: <nil> Duration: 6.149243769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X.] duration: 5.912446569s trace: https://jaeger.dev.storj.io/trace/38e62ebd31305c3a FAIL (2025-12-28 02:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA) context deadline exceeded ======LOGS====== c 28 01:59:54.632149533 (538h30m54.503232578s) [upload=472271]: Starting upload Dec 28 01:59:54.632223104 (538h30m54.503306149s) [upload=472271]: Done starting upload Dec 28 01:59:54.632459984 (538h30m54.50354303s) [upload=472271]: Uploading segments... Dec 28 01:59:54.632518817 (538h30m54.503601862s) [upload=472271]: Got next segment. Inline: false Dec 28 01:59:54.632550427 (538h30m54.503633462s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:59:54.632555987 (538h30m54.503639033s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:59:54.877485034 (538h30m54.748568069s) [seg_pos={0 0}, upload=472271]: Begin upload segment... Dec 28 01:59:54.877531954 (538h30m54.748614999s) [seg_pos={0 0}, upload=472271]: Done begin upload segment. Dec 28 01:59:54.877558324 (538h30m54.748641369s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.877632014 (538h30m54.748715059s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878258988 (538h30m54.749342033s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878542548 (538h30m54.749625583s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878774830 (538h30m54.749857865s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878917913 (538h30m54.750000958s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879363390 (538h30m54.750446425s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879398207 (538h30m54.750481242s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879728556 (538h30m54.750811591s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879962371 (538h30m54.751045406s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880052162 (538h30m54.751135197s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880298681 (538h30m54.751381717s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880517267 (538h30m54.751600302s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880627277 (538h30m54.751710322s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881125847 (538h30m54.752208892s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881996054 (538h30m54.753079089s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:56.553985411 (538h30m56.425068446s) [upload=472271]: Next returned nil segment Dec 28 01:59:56.554003425 (538h30m56.42508647s) [upload=472271]: Waiting for error group managing segments... Dec 28 01:59:56.879117778 (538h30m56.750200823s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.885385372 (538h30m56.756468407s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.890295382 (538h30m56.761378417s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.893360628 (538h30m56.764443663s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.896192852 (538h30m56.767275897s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.898752405 (538h30m56.76983544s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.900930874 (538h30m56.772013909s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.906041586 (538h30m56.777124621s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.908767286 (538h30m56.779850321s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.914066997 (538h30m56.785150032s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.160972856 (538h30m57.032055902s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.175758361 (538h30m57.046841396s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176365047 (538h30m57.047448082s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176385485 (538h30m57.04746852s) [seg_pos={0 0}, upload=472271]: Segment reached optimal threshold of 13 pieces. Dec 28 01:59:57.176565698 (538h30m57.047648733s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection Dec 28 01:59:57.176571319 (538h30m57.047654354s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176753717 (538h30m57.047836762s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176765208 (538h30m57.047848253s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176780998 (538h30m57.047864033s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176789875 (538h30m57.04787291s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176861181 (538h30m57.047944216s) [seg_pos={0 0}, upload=472271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:59:57.176948928 (538h30m57.048031963s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:59:57.176954178 (538h30m57.048037213s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:59:57.464405794 (538h30m57.335488839s) [upload=472271]: Done uploading segments... Dec 28 01:59:57.464430300 (538h30m57.335513336s) [upload=472271]: Upload finished. err: <nil> Duration: 5.91247342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X] duration: 30.00626836s trace: https://jaeger.dev.storj.io/trace/38e62ebd31305c3a FAIL (2025-12-28 02:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA) context deadline exceeded ======LOGS====== c 28 01:59:54.632149533 (538h30m54.503232578s) [upload=472271]: Starting upload Dec 28 01:59:54.632223104 (538h30m54.503306149s) [upload=472271]: Done starting upload Dec 28 01:59:54.632459984 (538h30m54.50354303s) [upload=472271]: Uploading segments... Dec 28 01:59:54.632518817 (538h30m54.503601862s) [upload=472271]: Got next segment. Inline: false Dec 28 01:59:54.632550427 (538h30m54.503633462s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:59:54.632555987 (538h30m54.503639033s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:59:54.877485034 (538h30m54.748568069s) [seg_pos={0 0}, upload=472271]: Begin upload segment... Dec 28 01:59:54.877531954 (538h30m54.748614999s) [seg_pos={0 0}, upload=472271]: Done begin upload segment. Dec 28 01:59:54.877558324 (538h30m54.748641369s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.877632014 (538h30m54.748715059s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878258988 (538h30m54.749342033s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878542548 (538h30m54.749625583s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878774830 (538h30m54.749857865s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.878917913 (538h30m54.750000958s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879363390 (538h30m54.750446425s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879398207 (538h30m54.750481242s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879728556 (538h30m54.750811591s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.879962371 (538h30m54.751045406s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880052162 (538h30m54.751135197s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880298681 (538h30m54.751381717s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880517267 (538h30m54.751600302s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.880627277 (538h30m54.751710322s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881125847 (538h30m54.752208892s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:54.881996054 (538h30m54.753079089s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Uploading piece... Dec 28 01:59:56.553985411 (538h30m56.425068446s) [upload=472271]: Next returned nil segment Dec 28 01:59:56.554003425 (538h30m56.42508647s) [upload=472271]: Waiting for error group managing segments... Dec 28 01:59:56.879117778 (538h30m56.750200823s) [noise=true , address=145.79.227.8:4331 , piece_id=INTSRNRD3FSGZI5VTJIROPIA5FHG4M5AZGBZEN5STC2YBWQX5V3A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.885385372 (538h30m56.756468407s) [noise=true , address=145.79.227.9:4365 , piece_id=A6UO22XXJQDZG5SOVJX7DWMOJBIXMUKIBOVQAMREVRXHSR33FZIQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.890295382 (538h30m56.761378417s) [noise=true , address=145.79.227.46:4601 , piece_id=ZUFAN5I2FC4O3PUP2V234RM6QMPP3TSVNVICDRNXYI6WDPHSHXNA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.893360628 (538h30m56.764443663s) [noise=true , address=145.79.227.7:4293 , piece_id=5JNV55DRIS4JZ7MSVQ4CZFPY6RODIXSZBAZW2JBDOS6HRK5TX7BQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.896192852 (538h30m56.767275897s) [noise=true , address=145.79.227.27:4691 , piece_id=TVDS7NLUW5F7UICAPIOQIFPKZTNMB4L6RTZLECHRXCPQCPWHZ2ZQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.898752405 (538h30m56.76983544s) [noise=true , address=145.79.227.48:4303 , piece_id=UIWMGA4RASDHJS6DTYDGZ7FT7JSXZPVVZOW34G36MA7J5XGNQFTQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.900930874 (538h30m56.772013909s) [noise=true , address=145.79.227.14:4838 , piece_id=C6GAZYDQMR53WCOOSAIOQAFSMOH2IVVXJNZOX42OT2E2WYRZHTJQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.906041586 (538h30m56.777124621s) [noise=true , address=145.79.227.17:4971 , piece_id=GLHK74JFQPJ3BSGB5QVUOH26IICQ4BOQPTIGWV4O4ECJ5VHPAFCA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.908767286 (538h30m56.779850321s) [noise=true , address=145.79.227.52:4446 , piece_id=MGJYGETMQKCCOJ2LNJ2OJGYCC4G2XIWAHUN5MO5R3546KLYJRZVQ, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:56.914066997 (538h30m56.785150032s) [noise=true , address=145.79.227.11:4730 , piece_id=BGQWKRKPQV3PHU46QWPR6I2KFTEO3556RMQFJQFU625UPU2ETW5A, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.160972856 (538h30m57.032055902s) [noise=true , address=45.91.116.12:4227 , piece_id=NT7GXP65DJWKVXYCS3LUMZLOSBXEJJDRQE7HNSJ2BZ4D25QHNAFA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.175758361 (538h30m57.046841396s) [noise=true , address=77.90.31.21:4880 , piece_id=KQB5MRGYMWHZ2WRNNQLOKWTRZ4ISNQSRAQN67JR77VQRN6676NTA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176365047 (538h30m57.047448082s) [noise=true , address=77.90.31.18:4805 , piece_id=K47V5KUAALJOXFCVQMYYRPX6DOMGTSVP5BDC25NKTGBFUYY2N6GA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: <nil> Dec 28 01:59:57.176385485 (538h30m57.04746852s) [seg_pos={0 0}, upload=472271]: Segment reached optimal threshold of 13 pieces. Dec 28 01:59:57.176565698 (538h30m57.047648733s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47806->77.90.31.19:4844: use of closed network connection Dec 28 01:59:57.176571319 (538h30m57.047654354s) [noise=true , address=77.90.31.19:4844 , piece_id=IXW2ESCLPN2SYV3OFKKZV7ZHEMFDMCKTPLZHPLXYSKEKZCZ6CVQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176753717 (538h30m57.047836762s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176765208 (538h30m57.047848253s) [noise=true , address=145.79.227.44:4193 , piece_id=UPCKISHAFJXK672IHP4A3RQTUYNKZO6HSMMLJBE76UUQMSD5EH5Q, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176780998 (538h30m57.047864033s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:59:57.176789875 (538h30m57.04787291s) [noise=true , address=45.91.116.6:4973 , piece_id=7JBR7XOCBMD5XBPWSDFUGIWQJCWUDQX356MG2EIGGJKMJZQ5AOQA, seg_pos={0 0}, upload=472271]: Long tail cancellation due to optimal threshold Dec 28 01:59:57.176861181 (538h30m57.047944216s) [seg_pos={0 0}, upload=472271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:59:57.176948928 (538h30m57.048031963s) [upload=472271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:59:57.176954178 (538h30m57.048037213s) [upload=472271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:59:57.464405794 (538h30m57.335488839s) [upload=472271]: Done uploading segments... Dec 28 01:59:57.464430300 (538h30m57.335513336s) [upload=472271]: Upload finished. err: <nil> Duration: 30.006293257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.] duration: 6.07978078s trace: https://jaeger.dev.storj.io/trace/54d414712f37049 FAIL (2025-12-28 01:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ) context deadline exceeded ======LOGS====== Dec 28 01:58:32.376916101 (538h29m32.247999136s) [upload=472250]: Starting upload Dec 28 01:58:32.376969372 (538h29m32.248052417s) [upload=472250]: Done starting upload Dec 28 01:58:32.377088629 (538h29m32.248171674s) [upload=472250]: Uploading segments... Dec 28 01:58:32.377186125 (538h29m32.24826917s) [upload=472250]: Got next segment. Inline: false Dec 28 01:58:32.377243905 (538h29m32.24832695s) [upload=472250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:58:32.377249004 (538h29m32.248332039s) [upload=472250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:58:32.627392098 (538h29m32.498475133s) [seg_pos={0 0}, upload=472250]: Begin upload segment... Dec 28 01:58:32.627425311 (538h29m32.498508346s) [seg_pos={0 0}, upload=472250]: Done begin upload segment. Dec 28 01:58:32.627455559 (538h29m32.498538594s) [noise=true , address=45.91.116.12:4227 , piece_id=M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.631319386 (538h29m32.502402421s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.632705165 (538h29m32.5037882s) [noise=true , address=145.79.227.45:4229 , piece_id=A7UKQ5GH4VWYIUVXGSC6EOSB2ROB7AE52TRRWJPJZXPJYBJ6AVTQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.632792521 (538h29m32.503875556s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.633872337 (538h29m32.504955372s) [noise=true , address=45.91.116.5:4299 , piece_id=LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.633956739 (538h29m32.505039784s) [noise=true , address=145.79.227.39:4013 , piece_id=VYSMJTCDGW3PHEUYIYTSLB6ZMGC33ZYVL7NB2C5E5QQMACST5UQA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.634919532 (538h29m32.506002567s) [noise=true , address=145.79.227.46:4603 , piece_id=23HHUGU4G7LO5SK3D3NF4LTG6STSIKFE52LZUMQKKUOT2AGNASMA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.635020764 (538h29m32.506103799s) [noise=true , address=145.79.227.6:4259 , piece_id=VEYWZGBZCM5W6SEKR6AWTVVA3QUD7OLUXBAXIXZK5FZV4FVQ5H4A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.636110710 (538h29m32.507193745s) [noise=true , address=145.79.227.8:4331 , piece_id=OO4DRPQAV6733EEAONRCJ3CFWY7T42T3RKMSHM7NFHLEGOA43P7A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.637060338 (538h29m32.508143373s) [noise=true , address=77.90.31.16:4735 , piece_id=QVBMAFFYSUBWJGIUX6GC37OMVS4AD2UHZIPN46NDEHZSINXB664A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.637634622 (538h29m32.508717657s) [noise=true , address=145.79.227.51:4409 , piece_id=ZU4KHXZ4AXSS53ABKFCJBYOYSFWUXW64R6FMKOZ23HWLGE6HYE7Q, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.638456507 (538h29m32.509539542s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.638773440 (538h29m32.509856475s) [noise=true , address=77.90.31.12:4590 , piece_id=PMRA6VZQDHF7W65RZNUYT2L6ZKMTX755MU2ZFY2RRWEAVHHIRVUQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.639626876 (538h29m32.510709911s) [noise=true , address=145.79.227.19:4079 , piece_id=7LOIX5XMQPNCVISNPKITW62NHNADDTVIT4PMUNJAN5ULZBMPSXAQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.640169077 (538h29m32.511252113s) [noise=true , address=77.90.31.10:4517 , piece_id=BIBFCUNNHC7XI43YZEU6TGFWLOWYYPTNOZMVYUJVLWSYZOQ4I4AA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.640840927 (538h29m32.511923973s) [noise=true , address=145.79.227.9:4365 , piece_id=SHRW3XIXFVXZEDDN52NXEAQFJPSZN2XQJWPASHRL36SUSFSJCGLA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:34.346305577 (538h29m34.217388612s) [upload=472250]: Next returned nil segment Dec 28 01:58:34.346321708 (538h29m34.217404743s) [upload=472250]: Waiting for error group managing segments... Dec 28 01:58:34.624853720 (538h29m34.495936755s) [noise=true , address=145.79.227.19:4079 , piece_id=7LOIX5XMQPNCVISNPKITW62NHNADDTVIT4PMUNJAN5ULZBMPSXAQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.627471835 (538h29m34.49855487s) [noise=true , address=145.79.227.45:4229 , piece_id=A7UKQ5GH4VWYIUVXGSC6EOSB2ROB7AE52TRRWJPJZXPJYBJ6AVTQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.632228503 (538h29m34.503311538s) [noise=true , address=145.79.227.8:4331 , piece_id=OO4DRPQAV6733EEAONRCJ3CFWY7T42T3RKMSHM7NFHLEGOA43P7A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.632298025 (538h29m34.503381061s) [noise=true , address=145.79.227.51:4409 , piece_id=ZU4KHXZ4AXSS53ABKFCJBYOYSFWUXW64R6FMKOZ23HWLGE6HYE7Q, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.648443208 (538h29m34.519526253s) [noise=true , address=145.79.227.9:4365 , piece_id=SHRW3XIXFVXZEDDN52NXEAQFJPSZN2XQJWPASHRL36SUSFSJCGLA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.654074201 (538h29m34.525157236s) [noise=true , address=145.79.227.39:4013 , piece_id=VYSMJTCDGW3PHEUYIYTSLB6ZMGC33ZYVL7NB2C5E5QQMACST5UQA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.656169029 (538h29m34.527252064s) [noise=true , address=145.79.227.46:4603 , piece_id=23HHUGU4G7LO5SK3D3NF4LTG6STSIKFE52LZUMQKKUOT2AGNASMA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.736754027 (538h29m34.607837062s) [noise=true , address=145.79.227.6:4259 , piece_id=VEYWZGBZCM5W6SEKR6AWTVVA3QUD7OLUXBAXIXZK5FZV4FVQ5H4A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.888399856 (538h29m34.759482901s) [noise=true , address=77.90.31.12:4590 , piece_id=PMRA6VZQDHF7W65RZNUYT2L6ZKMTX755MU2ZFY2RRWEAVHHIRVUQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.902578814 (538h29m34.77366186s) [noise=true , address=77.90.31.10:4517 , piece_id=BIBFCUNNHC7XI43YZEU6TGFWLOWYYPTNOZMVYUJVLWSYZOQ4I4AA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.550357033 (538h29m35.421440079s) [noise=true , address=45.91.116.12:4227 , piece_id=M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.893728039 (538h29m35.764811084s) [noise=true , address=45.91.116.5:4299 , piece_id=LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.929747616 (538h29m35.800830661s) [noise=true , address=77.90.31.16:4735 , piece_id=QVBMAFFYSUBWJGIUX6GC37OMVS4AD2UHZIPN46NDEHZSINXB664A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.929774287 (538h29m35.800857332s) [seg_pos={0 0}, upload=472250]: Segment reached optimal threshold of 13 pieces. Dec 28 01:58:35.930044331 (538h29m35.801127366s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930061053 (538h29m35.801144088s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930287835 (538h29m35.80137088s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930338070 (538h29m35.801421125s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930349964 (538h29m35.801432999s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930358269 (538h29m35.801441304s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930379950 (538h29m35.801462985s) [seg_pos={0 0}, upload=472250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:58:35.930457649 (538h29m35.801540694s) [upload=472250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:58:35.930462458 (538h29m35.801545493s) [upload=472250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:58:36.224914513 (538h29m36.095997549s) [upload=472250]: Done uploading segments... Dec 28 01:58:36.224933370 (538h29m36.096016415s) [upload=472250]: Upload finished. err: <nil> Duration: 6.079806699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X] duration: 30.00417381s trace: https://jaeger.dev.storj.io/trace/54d414712f37049 FAIL (2025-12-28 01:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ) context deadline exceeded ======LOGS====== Dec 28 01:58:32.376916101 (538h29m32.247999136s) [upload=472250]: Starting upload Dec 28 01:58:32.376969372 (538h29m32.248052417s) [upload=472250]: Done starting upload Dec 28 01:58:32.377088629 (538h29m32.248171674s) [upload=472250]: Uploading segments... Dec 28 01:58:32.377186125 (538h29m32.24826917s) [upload=472250]: Got next segment. Inline: false Dec 28 01:58:32.377243905 (538h29m32.24832695s) [upload=472250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:58:32.377249004 (538h29m32.248332039s) [upload=472250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:58:32.627392098 (538h29m32.498475133s) [seg_pos={0 0}, upload=472250]: Begin upload segment... Dec 28 01:58:32.627425311 (538h29m32.498508346s) [seg_pos={0 0}, upload=472250]: Done begin upload segment. Dec 28 01:58:32.627455559 (538h29m32.498538594s) [noise=true , address=45.91.116.12:4227 , piece_id=M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.631319386 (538h29m32.502402421s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.632705165 (538h29m32.5037882s) [noise=true , address=145.79.227.45:4229 , piece_id=A7UKQ5GH4VWYIUVXGSC6EOSB2ROB7AE52TRRWJPJZXPJYBJ6AVTQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.632792521 (538h29m32.503875556s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.633872337 (538h29m32.504955372s) [noise=true , address=45.91.116.5:4299 , piece_id=LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.633956739 (538h29m32.505039784s) [noise=true , address=145.79.227.39:4013 , piece_id=VYSMJTCDGW3PHEUYIYTSLB6ZMGC33ZYVL7NB2C5E5QQMACST5UQA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.634919532 (538h29m32.506002567s) [noise=true , address=145.79.227.46:4603 , piece_id=23HHUGU4G7LO5SK3D3NF4LTG6STSIKFE52LZUMQKKUOT2AGNASMA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.635020764 (538h29m32.506103799s) [noise=true , address=145.79.227.6:4259 , piece_id=VEYWZGBZCM5W6SEKR6AWTVVA3QUD7OLUXBAXIXZK5FZV4FVQ5H4A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.636110710 (538h29m32.507193745s) [noise=true , address=145.79.227.8:4331 , piece_id=OO4DRPQAV6733EEAONRCJ3CFWY7T42T3RKMSHM7NFHLEGOA43P7A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.637060338 (538h29m32.508143373s) [noise=true , address=77.90.31.16:4735 , piece_id=QVBMAFFYSUBWJGIUX6GC37OMVS4AD2UHZIPN46NDEHZSINXB664A, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.637634622 (538h29m32.508717657s) [noise=true , address=145.79.227.51:4409 , piece_id=ZU4KHXZ4AXSS53ABKFCJBYOYSFWUXW64R6FMKOZ23HWLGE6HYE7Q, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.638456507 (538h29m32.509539542s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.638773440 (538h29m32.509856475s) [noise=true , address=77.90.31.12:4590 , piece_id=PMRA6VZQDHF7W65RZNUYT2L6ZKMTX755MU2ZFY2RRWEAVHHIRVUQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.639626876 (538h29m32.510709911s) [noise=true , address=145.79.227.19:4079 , piece_id=7LOIX5XMQPNCVISNPKITW62NHNADDTVIT4PMUNJAN5ULZBMPSXAQ, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.640169077 (538h29m32.511252113s) [noise=true , address=77.90.31.10:4517 , piece_id=BIBFCUNNHC7XI43YZEU6TGFWLOWYYPTNOZMVYUJVLWSYZOQ4I4AA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:32.640840927 (538h29m32.511923973s) [noise=true , address=145.79.227.9:4365 , piece_id=SHRW3XIXFVXZEDDN52NXEAQFJPSZN2XQJWPASHRL36SUSFSJCGLA, seg_pos={0 0}, upload=472250]: Uploading piece... Dec 28 01:58:34.346305577 (538h29m34.217388612s) [upload=472250]: Next returned nil segment Dec 28 01:58:34.346321708 (538h29m34.217404743s) [upload=472250]: Waiting for error group managing segments... Dec 28 01:58:34.624853720 (538h29m34.495936755s) [noise=true , address=145.79.227.19:4079 , piece_id=7LOIX5XMQPNCVISNPKITW62NHNADDTVIT4PMUNJAN5ULZBMPSXAQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.627471835 (538h29m34.49855487s) [noise=true , address=145.79.227.45:4229 , piece_id=A7UKQ5GH4VWYIUVXGSC6EOSB2ROB7AE52TRRWJPJZXPJYBJ6AVTQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.632228503 (538h29m34.503311538s) [noise=true , address=145.79.227.8:4331 , piece_id=OO4DRPQAV6733EEAONRCJ3CFWY7T42T3RKMSHM7NFHLEGOA43P7A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.632298025 (538h29m34.503381061s) [noise=true , address=145.79.227.51:4409 , piece_id=ZU4KHXZ4AXSS53ABKFCJBYOYSFWUXW64R6FMKOZ23HWLGE6HYE7Q, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.648443208 (538h29m34.519526253s) [noise=true , address=145.79.227.9:4365 , piece_id=SHRW3XIXFVXZEDDN52NXEAQFJPSZN2XQJWPASHRL36SUSFSJCGLA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.654074201 (538h29m34.525157236s) [noise=true , address=145.79.227.39:4013 , piece_id=VYSMJTCDGW3PHEUYIYTSLB6ZMGC33ZYVL7NB2C5E5QQMACST5UQA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.656169029 (538h29m34.527252064s) [noise=true , address=145.79.227.46:4603 , piece_id=23HHUGU4G7LO5SK3D3NF4LTG6STSIKFE52LZUMQKKUOT2AGNASMA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.736754027 (538h29m34.607837062s) [noise=true , address=145.79.227.6:4259 , piece_id=VEYWZGBZCM5W6SEKR6AWTVVA3QUD7OLUXBAXIXZK5FZV4FVQ5H4A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.888399856 (538h29m34.759482901s) [noise=true , address=77.90.31.12:4590 , piece_id=PMRA6VZQDHF7W65RZNUYT2L6ZKMTX755MU2ZFY2RRWEAVHHIRVUQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:34.902578814 (538h29m34.77366186s) [noise=true , address=77.90.31.10:4517 , piece_id=BIBFCUNNHC7XI43YZEU6TGFWLOWYYPTNOZMVYUJVLWSYZOQ4I4AA, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.550357033 (538h29m35.421440079s) [noise=true , address=45.91.116.12:4227 , piece_id=M3MLQ2YPF27BU2RVD67PN47JIJQQGL7YROOQ6U4VL3U33NXDFQUQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.893728039 (538h29m35.764811084s) [noise=true , address=45.91.116.5:4299 , piece_id=LKDOXJ3OI6E7X32XAQGY4EUJSMSOAU7SK7CRKYF3KMDXVOJUEU7Q, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.929747616 (538h29m35.800830661s) [noise=true , address=77.90.31.16:4735 , piece_id=QVBMAFFYSUBWJGIUX6GC37OMVS4AD2UHZIPN46NDEHZSINXB664A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: <nil> Dec 28 01:58:35.929774287 (538h29m35.800857332s) [seg_pos={0 0}, upload=472250]: Segment reached optimal threshold of 13 pieces. Dec 28 01:58:35.930044331 (538h29m35.801127366s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930061053 (538h29m35.801144088s) [noise=true , address=145.79.227.48:4301 , piece_id=BR7INIY32Y3GHBTWORFZV665QO5KUZ7JIVE5WWGZBXC4NHG5NVPQ, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930287835 (538h29m35.80137088s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930338070 (538h29m35.801421125s) [noise=true , address=45.91.116.11:4119 , piece_id=OBYQJ6XYYMO3M4P3E2GIZSGWA6GF7UZFSKJE3AEF6J5INIOHEC3A, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930349964 (538h29m35.801432999s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 01:58:35.930358269 (538h29m35.801441304s) [noise=true , address=145.79.227.34:4905 , piece_id=KWYX2QDXUGR6BG77K7BH4QOZ2J7QCNXKD3SW74DATSDI7TU5HIEQ, seg_pos={0 0}, upload=472250]: Long tail cancellation due to optimal threshold Dec 28 01:58:35.930379950 (538h29m35.801462985s) [seg_pos={0 0}, upload=472250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:58:35.930457649 (538h29m35.801540694s) [upload=472250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:58:35.930462458 (538h29m35.801545493s) [upload=472250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:58:36.224914513 (538h29m36.095997549s) [upload=472250]: Done uploading segments... Dec 28 01:58:36.224933370 (538h29m36.096016415s) [upload=472250]: Upload finished. err: <nil> Duration: 30.004200009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.] duration: 6.692747669s trace: https://jaeger.dev.storj.io/trace/813e48fdd7d5992 FAIL (2025-12-28 01:57:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, Piece ID: SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ) context deadline exceeded ======LOGS====== Dec 28 01:57:12.966064825 (538h28m12.83714787s) [upload=472232]: Starting upload Dec 28 01:57:12.966147402 (538h28m12.837230458s) [upload=472232]: Done starting upload Dec 28 01:57:12.966223367 (538h28m12.837306412s) [upload=472232]: Uploading segments... Dec 28 01:57:12.966450439 (538h28m12.837533474s) [upload=472232]: Got next segment. Inline: false Dec 28 01:57:12.966484464 (538h28m12.837567499s) [upload=472232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:57:12.966489915 (538h28m12.83757295s) [upload=472232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:57:13.213428668 (538h28m13.084511703s) [seg_pos={0 0}, upload=472232]: Begin upload segment... Dec 28 01:57:13.213496808 (538h28m13.084579843s) [seg_pos={0 0}, upload=472232]: Done begin upload segment. Dec 28 01:57:13.213508931 (538h28m13.084591966s) [noise=true , address=77.90.31.17:4770 , piece_id=J7QPGR7CFXV6ERJZVOZECUZPR4TNQYB5A4CBLMQH5OMAKUAF3BNQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.213543637 (538h28m13.084626672s) [noise=true , address=45.91.116.13:4154 , piece_id=T7DV2XG2CSJMS4W5JB5FGQB6WSKJURJHQ72LLBCSQULJGCT54EIQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.213981301 (538h28m13.085064336s) [noise=true , address=145.79.227.5:4221 , piece_id=2KPSXR3ILGZOJKECDJ3ETG6IQID3JC4V3JGC3KJR7PMG7DQRJXPQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214258178 (538h28m13.085341223s) [noise=true , address=145.79.227.16:4912 , piece_id=XTAB2BLNKYI73B6MHQOLAAYQPNVDM7GPIC2TCMLUUWTRVNFE33FA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214541026 (538h28m13.085624061s) [noise=true , address=145.79.227.15:4875 , piece_id=LQPNOVTFI3D2FMXQWVITRBGNPABG2U7Y77BTH42BWWT46WJZ7EOQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214956417 (538h28m13.086039452s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215256638 (538h28m13.086339673s) [noise=true , address=45.91.116.6:4974 , piece_id=OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215586307 (538h28m13.086669342s) [noise=true , address=145.79.227.17:4972 , piece_id=SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215902739 (538h28m13.086985774s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.216252635 (538h28m13.08733567s) [noise=true , address=145.79.227.31:4798 , piece_id=WVDL6BXV4H53BOU6E2UPL3OIEXXAHFNWMEM2DYM47XFEFETNYAXA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.217348271 (538h28m13.088431316s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.217469863 (538h28m13.088552908s) [noise=true , address=77.90.31.19:4844 , piece_id=P4ZRBWMVBNCDUWDLFNSR4DADUW2BE7ZXZA3RANH5G4XIJ7B52BZA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.218490687 (538h28m13.089573722s) [noise=true , address=145.79.227.52:4446 , piece_id=6ZAK77MFAOH3RZ5UKBCUXATUWQ6VQN2GCFZG6L7QFLDQSMABCGQA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.222714701 (538h28m13.093797736s) [noise=true , address=145.79.227.9:4365 , piece_id=TH5AOXAUMNFGLEF3BLI7IEAMJSXVCIH5YSLEI6AHZGTWRMOWYZFA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.225415924 (538h28m13.096498969s) [noise=true , address=45.91.116.14:4189 , piece_id=YG6IVQSDFIOAQ3VVKQAWXP2P6PCU3V4OUCCIZBXEHY4SYGLOBM2A, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.227791347 (538h28m13.098874392s) [noise=true , address=145.79.227.42:4123 , piece_id=BUQJJ2ERVQE2TXBWT6NVSFV5ZCOGLVM65R33ZTZGT2MQ7D7DVGAQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:14.934453614 (538h28m14.805536649s) [upload=472232]: Next returned nil segment Dec 28 01:57:14.934469225 (538h28m14.80555226s) [upload=472232]: Waiting for error group managing segments... Dec 28 01:57:15.226285662 (538h28m15.097368698s) [noise=true , address=145.79.227.16:4912 , piece_id=XTAB2BLNKYI73B6MHQOLAAYQPNVDM7GPIC2TCMLUUWTRVNFE33FA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.229278912 (538h28m15.100361947s) [noise=true , address=145.79.227.17:4972 , piece_id=SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.235964593 (538h28m15.107047628s) [noise=true , address=145.79.227.5:4221 , piece_id=2KPSXR3ILGZOJKECDJ3ETG6IQID3JC4V3JGC3KJR7PMG7DQRJXPQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.238574112 (538h28m15.109657147s) [noise=true , address=145.79.227.31:4798 , piece_id=WVDL6BXV4H53BOU6E2UPL3OIEXXAHFNWMEM2DYM47XFEFETNYAXA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.241069404 (538h28m15.112152449s) [noise=true , address=145.79.227.9:4365 , piece_id=TH5AOXAUMNFGLEF3BLI7IEAMJSXVCIH5YSLEI6AHZGTWRMOWYZFA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.245159581 (538h28m15.116242617s) [noise=true , address=145.79.227.42:4123 , piece_id=BUQJJ2ERVQE2TXBWT6NVSFV5ZCOGLVM65R33ZTZGT2MQ7D7DVGAQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.245479802 (538h28m15.116562857s) [noise=true , address=145.79.227.52:4446 , piece_id=6ZAK77MFAOH3RZ5UKBCUXATUWQ6VQN2GCFZG6L7QFLDQSMABCGQA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.301032985 (538h28m15.17211602s) [noise=true , address=145.79.227.15:4875 , piece_id=LQPNOVTFI3D2FMXQWVITRBGNPABG2U7Y77BTH42BWWT46WJZ7EOQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.421871472 (538h28m15.292954507s) [noise=true , address=77.90.31.17:4770 , piece_id=J7QPGR7CFXV6ERJZVOZECUZPR4TNQYB5A4CBLMQH5OMAKUAF3BNQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.458390921 (538h28m15.329473956s) [noise=true , address=45.91.116.13:4154 , piece_id=T7DV2XG2CSJMS4W5JB5FGQB6WSKJURJHQ72LLBCSQULJGCT54EIQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.461611423 (538h28m15.332694458s) [noise=true , address=45.91.116.14:4189 , piece_id=YG6IVQSDFIOAQ3VVKQAWXP2P6PCU3V4OUCCIZBXEHY4SYGLOBM2A, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.501442879 (538h28m15.372525914s) [noise=true , address=45.91.116.6:4974 , piece_id=OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.536208599 (538h28m15.407291644s) [noise=true , address=77.90.31.19:4844 , piece_id=P4ZRBWMVBNCDUWDLFNSR4DADUW2BE7ZXZA3RANH5G4XIJ7B52BZA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.536287179 (538h28m15.407370224s) [seg_pos={0 0}, upload=472232]: Segment reached optimal threshold of 13 pieces. Dec 28 01:57:15.536567983 (538h28m15.407651038s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 28 01:57:15.536577292 (538h28m15.407660337s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.536827668 (538h28m15.407910713s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection Dec 28 01:57:15.536836535 (538h28m15.40791958s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.537216759 (538h28m15.408299814s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: context canceled; context canceled Dec 28 01:57:15.537227619 (538h28m15.408310664s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.537251986 (538h28m15.408335031s) [seg_pos={0 0}, upload=472232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:57:15.537360282 (538h28m15.408443317s) [upload=472232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:57:15.537367005 (538h28m15.40845006s) [upload=472232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:57:15.823054563 (538h28m15.694137608s) [upload=472232]: Done uploading segments... Dec 28 01:57:15.823078569 (538h28m15.694161604s) [upload=472232]: Upload finished. err: <nil> Duration: 6.692762336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X] duration: 30.003169123s trace: https://jaeger.dev.storj.io/trace/813e48fdd7d5992 FAIL (2025-12-28 01:57:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, Piece ID: SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ) context deadline exceeded ======LOGS====== Dec 28 01:57:12.966064825 (538h28m12.83714787s) [upload=472232]: Starting upload Dec 28 01:57:12.966147402 (538h28m12.837230458s) [upload=472232]: Done starting upload Dec 28 01:57:12.966223367 (538h28m12.837306412s) [upload=472232]: Uploading segments... Dec 28 01:57:12.966450439 (538h28m12.837533474s) [upload=472232]: Got next segment. Inline: false Dec 28 01:57:12.966484464 (538h28m12.837567499s) [upload=472232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:57:12.966489915 (538h28m12.83757295s) [upload=472232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:57:13.213428668 (538h28m13.084511703s) [seg_pos={0 0}, upload=472232]: Begin upload segment... Dec 28 01:57:13.213496808 (538h28m13.084579843s) [seg_pos={0 0}, upload=472232]: Done begin upload segment. Dec 28 01:57:13.213508931 (538h28m13.084591966s) [noise=true , address=77.90.31.17:4770 , piece_id=J7QPGR7CFXV6ERJZVOZECUZPR4TNQYB5A4CBLMQH5OMAKUAF3BNQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.213543637 (538h28m13.084626672s) [noise=true , address=45.91.116.13:4154 , piece_id=T7DV2XG2CSJMS4W5JB5FGQB6WSKJURJHQ72LLBCSQULJGCT54EIQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.213981301 (538h28m13.085064336s) [noise=true , address=145.79.227.5:4221 , piece_id=2KPSXR3ILGZOJKECDJ3ETG6IQID3JC4V3JGC3KJR7PMG7DQRJXPQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214258178 (538h28m13.085341223s) [noise=true , address=145.79.227.16:4912 , piece_id=XTAB2BLNKYI73B6MHQOLAAYQPNVDM7GPIC2TCMLUUWTRVNFE33FA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214541026 (538h28m13.085624061s) [noise=true , address=145.79.227.15:4875 , piece_id=LQPNOVTFI3D2FMXQWVITRBGNPABG2U7Y77BTH42BWWT46WJZ7EOQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.214956417 (538h28m13.086039452s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215256638 (538h28m13.086339673s) [noise=true , address=45.91.116.6:4974 , piece_id=OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215586307 (538h28m13.086669342s) [noise=true , address=145.79.227.17:4972 , piece_id=SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.215902739 (538h28m13.086985774s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.216252635 (538h28m13.08733567s) [noise=true , address=145.79.227.31:4798 , piece_id=WVDL6BXV4H53BOU6E2UPL3OIEXXAHFNWMEM2DYM47XFEFETNYAXA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.217348271 (538h28m13.088431316s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.217469863 (538h28m13.088552908s) [noise=true , address=77.90.31.19:4844 , piece_id=P4ZRBWMVBNCDUWDLFNSR4DADUW2BE7ZXZA3RANH5G4XIJ7B52BZA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.218490687 (538h28m13.089573722s) [noise=true , address=145.79.227.52:4446 , piece_id=6ZAK77MFAOH3RZ5UKBCUXATUWQ6VQN2GCFZG6L7QFLDQSMABCGQA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.222714701 (538h28m13.093797736s) [noise=true , address=145.79.227.9:4365 , piece_id=TH5AOXAUMNFGLEF3BLI7IEAMJSXVCIH5YSLEI6AHZGTWRMOWYZFA, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.225415924 (538h28m13.096498969s) [noise=true , address=45.91.116.14:4189 , piece_id=YG6IVQSDFIOAQ3VVKQAWXP2P6PCU3V4OUCCIZBXEHY4SYGLOBM2A, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:13.227791347 (538h28m13.098874392s) [noise=true , address=145.79.227.42:4123 , piece_id=BUQJJ2ERVQE2TXBWT6NVSFV5ZCOGLVM65R33ZTZGT2MQ7D7DVGAQ, seg_pos={0 0}, upload=472232]: Uploading piece... Dec 28 01:57:14.934453614 (538h28m14.805536649s) [upload=472232]: Next returned nil segment Dec 28 01:57:14.934469225 (538h28m14.80555226s) [upload=472232]: Waiting for error group managing segments... Dec 28 01:57:15.226285662 (538h28m15.097368698s) [noise=true , address=145.79.227.16:4912 , piece_id=XTAB2BLNKYI73B6MHQOLAAYQPNVDM7GPIC2TCMLUUWTRVNFE33FA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.229278912 (538h28m15.100361947s) [noise=true , address=145.79.227.17:4972 , piece_id=SPYADR7RD7ZBGCZ4KJLNKWJV2U4VJS6MEVLHPOSFEM7W7IX7IDKA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.235964593 (538h28m15.107047628s) [noise=true , address=145.79.227.5:4221 , piece_id=2KPSXR3ILGZOJKECDJ3ETG6IQID3JC4V3JGC3KJR7PMG7DQRJXPQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.238574112 (538h28m15.109657147s) [noise=true , address=145.79.227.31:4798 , piece_id=WVDL6BXV4H53BOU6E2UPL3OIEXXAHFNWMEM2DYM47XFEFETNYAXA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.241069404 (538h28m15.112152449s) [noise=true , address=145.79.227.9:4365 , piece_id=TH5AOXAUMNFGLEF3BLI7IEAMJSXVCIH5YSLEI6AHZGTWRMOWYZFA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.245159581 (538h28m15.116242617s) [noise=true , address=145.79.227.42:4123 , piece_id=BUQJJ2ERVQE2TXBWT6NVSFV5ZCOGLVM65R33ZTZGT2MQ7D7DVGAQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.245479802 (538h28m15.116562857s) [noise=true , address=145.79.227.52:4446 , piece_id=6ZAK77MFAOH3RZ5UKBCUXATUWQ6VQN2GCFZG6L7QFLDQSMABCGQA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.301032985 (538h28m15.17211602s) [noise=true , address=145.79.227.15:4875 , piece_id=LQPNOVTFI3D2FMXQWVITRBGNPABG2U7Y77BTH42BWWT46WJZ7EOQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.421871472 (538h28m15.292954507s) [noise=true , address=77.90.31.17:4770 , piece_id=J7QPGR7CFXV6ERJZVOZECUZPR4TNQYB5A4CBLMQH5OMAKUAF3BNQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.458390921 (538h28m15.329473956s) [noise=true , address=45.91.116.13:4154 , piece_id=T7DV2XG2CSJMS4W5JB5FGQB6WSKJURJHQ72LLBCSQULJGCT54EIQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.461611423 (538h28m15.332694458s) [noise=true , address=45.91.116.14:4189 , piece_id=YG6IVQSDFIOAQ3VVKQAWXP2P6PCU3V4OUCCIZBXEHY4SYGLOBM2A, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.501442879 (538h28m15.372525914s) [noise=true , address=45.91.116.6:4974 , piece_id=OASGV6LNZVQVTJSCE7NZQFJ4XVCZNXZ6RCFQ6A7FQI5K5VSS63ZQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.536208599 (538h28m15.407291644s) [noise=true , address=77.90.31.19:4844 , piece_id=P4ZRBWMVBNCDUWDLFNSR4DADUW2BE7ZXZA3RANH5G4XIJ7B52BZA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: <nil> Dec 28 01:57:15.536287179 (538h28m15.407370224s) [seg_pos={0 0}, upload=472232]: Segment reached optimal threshold of 13 pieces. Dec 28 01:57:15.536567983 (538h28m15.407651038s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 28 01:57:15.536577292 (538h28m15.407660337s) [noise=true , address=145.79.227.50:4373 , piece_id=PRSPE77WN4ZC57KWHWW3FCIDVYJ2RKLNWPKWJN6AB7HFYZCA6MVA, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.536827668 (538h28m15.407910713s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38048->145.79.227.41:4087: use of closed network connection Dec 28 01:57:15.536836535 (538h28m15.40791958s) [noise=true , address=145.79.227.41:4087 , piece_id=VSG7JZ3MBREKNDH76CB6MAEFZI7HQDX3PAOAQZGETV2CYY2JVNEQ, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.537216759 (538h28m15.408299814s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: context canceled; context canceled Dec 28 01:57:15.537227619 (538h28m15.408310664s) [noise=true , address=45.91.116.9:4264 , piece_id=7V46ZRTNLLYFHXFJTAUEAJ4VQ5CEX45EZHHCZQDSEMFTGFZ5GS3A, seg_pos={0 0}, upload=472232]: Long tail cancellation due to optimal threshold Dec 28 01:57:15.537251986 (538h28m15.408335031s) [seg_pos={0 0}, upload=472232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:57:15.537360282 (538h28m15.408443317s) [upload=472232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:57:15.537367005 (538h28m15.40845006s) [upload=472232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:57:15.823054563 (538h28m15.694137608s) [upload=472232]: Done uploading segments... Dec 28 01:57:15.823078569 (538h28m15.694161604s) [upload=472232]: Upload finished. err: <nil> Duration: 30.003189851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.] duration: 22.250883822s trace: https://jaeger.dev.storj.io/trace/17477699a95ae3d4 FAIL (2025-12-28 01:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA) context deadline exceeded ======LOGS====== 01:55:19.863730382 (538h26m19.734813417s) [upload=472203]: Starting upload Dec 28 01:55:19.863811446 (538h26m19.734894481s) [upload=472203]: Done starting upload Dec 28 01:55:19.863916906 (538h26m19.734999941s) [upload=472203]: Uploading segments... Dec 28 01:55:19.863993402 (538h26m19.735076457s) [upload=472203]: Got next segment. Inline: false Dec 28 01:55:19.864031194 (538h26m19.735114229s) [upload=472203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:55:19.864036605 (538h26m19.73511964s) [upload=472203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:55:20.110407808 (538h26m19.981490843s) [seg_pos={0 0}, upload=472203]: Begin upload segment... Dec 28 01:55:20.110461169 (538h26m19.981544204s) [seg_pos={0 0}, upload=472203]: Done begin upload segment. Dec 28 01:55:20.110508519 (538h26m19.981591564s) [noise=true , address=145.79.227.36:4517 , piece_id=4HZU2GVP42BUXOC7I4BPGKMWDTY62GBF2U62T7WZIM25RY4622JA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.111363868 (538h26m19.982446903s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.111852699 (538h26m19.982935734s) [noise=true , address=45.91.116.5:4297 , piece_id=DJYTQCTMOUAJNCYXNEPUBBS7NC6L4CEPN2O5PZC2ZXOGAYBJ7KEQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.112289511 (538h26m19.983372546s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.112734107 (538h26m19.983817152s) [noise=true , address=145.79.227.11:4731 , piece_id=VCTCPEGZGZ36LRKYWJGBFZFAENPNQRKNU6CR47W2FH6S56DSRAUA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.113170789 (538h26m19.984253824s) [noise=true , address=145.79.227.46:4602 , piece_id=GDZWPYZFGJEO5QSP2GYVCWA7M2XFKVHSAJ7V2V2NWIAPI42CBPQQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.113535933 (538h26m19.984618968s) [noise=true , address=145.79.227.19:4077 , piece_id=VIVYUOXLZ3LMPSKWGMOSETNGJWCKFEEUR3V5COIJBJ34RG5G6OVQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.115039998 (538h26m19.986123033s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.116762938 (538h26m19.987845984s) [noise=true , address=45.91.116.6:4976 , piece_id=JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.119031007 (538h26m19.990114042s) [noise=true , address=45.91.116.8:4047 , piece_id=AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.120729502 (538h26m19.991812537s) [noise=true , address=145.79.227.9:4367 , piece_id=KHAKD3FVZGBMG23B7AHJNCHGCN4BCO2JCYPM3LDZB75K72NIB64A, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.121030786 (538h26m19.992113821s) [noise=true , address=145.79.227.3:4150 , piece_id=A4PNA3F62NEBXREBHY7WECL4U7CAKLQMJDJOWTRNUUZPN5UGJXLQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.122758064 (538h26m19.993841109s) [noise=true , address=145.79.227.51:4411 , piece_id=V2RTOKKPADMFYFDKQU5NNH2VY6O475A45PGPDZEKBBHHES6MA7OA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.124502676 (538h26m19.995585721s) [noise=true , address=145.79.227.15:4874 , piece_id=E7OWZJ3XIK2S5H7ZKNRROSIPYRUA4ULA4SG6RZPLXMYDUV7NLIPQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.124782760 (538h26m19.995865795s) [noise=true , address=145.79.227.31:4798 , piece_id=TRP43YNIWDSRTICXPRUEPMH2CFMXRHCVMAGGKHU7BNXYCFASKKTQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.127995918 (538h26m19.999078963s) [noise=true , address=45.91.116.14:4191 , piece_id=V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:21.841594073 (538h26m21.712677108s) [upload=472203]: Next returned nil segment Dec 28 01:55:21.841607999 (538h26m21.712691034s) [upload=472203]: Waiting for error group managing segments... Dec 28 01:55:22.131885026 (538h26m22.002968061s) [noise=true , address=145.79.227.51:4411 , piece_id=V2RTOKKPADMFYFDKQU5NNH2VY6O475A45PGPDZEKBBHHES6MA7OA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.132670642 (538h26m22.003753677s) [noise=true , address=145.79.227.3:4150 , piece_id=A4PNA3F62NEBXREBHY7WECL4U7CAKLQMJDJOWTRNUUZPN5UGJXLQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.135529235 (538h26m22.00661227s) [noise=true , address=145.79.227.31:4798 , piece_id=TRP43YNIWDSRTICXPRUEPMH2CFMXRHCVMAGGKHU7BNXYCFASKKTQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.143997921 (538h26m22.015080966s) [noise=true , address=145.79.227.46:4602 , piece_id=GDZWPYZFGJEO5QSP2GYVCWA7M2XFKVHSAJ7V2V2NWIAPI42CBPQQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.171746960 (538h26m22.042829995s) [noise=true , address=145.79.227.36:4517 , piece_id=4HZU2GVP42BUXOC7I4BPGKMWDTY62GBF2U62T7WZIM25RY4622JA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.177885619 (538h26m22.048968655s) [noise=true , address=145.79.227.9:4367 , piece_id=KHAKD3FVZGBMG23B7AHJNCHGCN4BCO2JCYPM3LDZB75K72NIB64A, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.192891123 (538h26m22.063974158s) [noise=true , address=145.79.227.11:4731 , piece_id=VCTCPEGZGZ36LRKYWJGBFZFAENPNQRKNU6CR47W2FH6S56DSRAUA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.195234274 (538h26m22.066317309s) [noise=true , address=145.79.227.19:4077 , piece_id=VIVYUOXLZ3LMPSKWGMOSETNGJWCKFEEUR3V5COIJBJ34RG5G6OVQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.214051786 (538h26m22.085134821s) [noise=true , address=145.79.227.15:4874 , piece_id=E7OWZJ3XIK2S5H7ZKNRROSIPYRUA4ULA4SG6RZPLXMYDUV7NLIPQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.364054808 (538h26m22.235137843s) [noise=true , address=45.91.116.8:4047 , piece_id=AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.399898760 (538h26m22.270981795s) [noise=true , address=45.91.116.14:4191 , piece_id=V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.419378313 (538h26m22.290461348s) [noise=true , address=45.91.116.6:4976 , piece_id=JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.511193444 (538h26m22.382276479s) [noise=true , address=45.91.116.5:4297 , piece_id=DJYTQCTMOUAJNCYXNEPUBBS7NC6L4CEPN2O5PZC2ZXOGAYBJ7KEQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.511214024 (538h26m22.382297059s) [seg_pos={0 0}, upload=472203]: Segment reached optimal threshold of 13 pieces. Dec 28 01:55:22.511382495 (538h26m22.38246553s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection Dec 28 01:55:22.511387975 (538h26m22.38247101s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511578598 (538h26m22.382661633s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 01:55:22.511584520 (538h26m22.382667565s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511721460 (538h26m22.382804495s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:55:22.511754663 (538h26m22.382837718s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511787535 (538h26m22.382870571s) [seg_pos={0 0}, upload=472203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:55:22.511947380 (538h26m22.383030425s) [upload=472203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:55:22.511955776 (538h26m22.383038821s) [upload=472203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:55:22.829773646 (538h26m22.700856691s) [upload=472203]: Done uploading segments... Dec 28 01:55:22.829829423 (538h26m22.700912488s) [upload=472203]: Upload finished. err: <nil> Duration: 22.250909439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X] duration: 30.00319408s trace: https://jaeger.dev.storj.io/trace/17477699a95ae3d4 FAIL (2025-12-28 01:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA) context deadline exceeded ======LOGS====== 01:55:19.863730382 (538h26m19.734813417s) [upload=472203]: Starting upload Dec 28 01:55:19.863811446 (538h26m19.734894481s) [upload=472203]: Done starting upload Dec 28 01:55:19.863916906 (538h26m19.734999941s) [upload=472203]: Uploading segments... Dec 28 01:55:19.863993402 (538h26m19.735076457s) [upload=472203]: Got next segment. Inline: false Dec 28 01:55:19.864031194 (538h26m19.735114229s) [upload=472203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:55:19.864036605 (538h26m19.73511964s) [upload=472203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:55:20.110407808 (538h26m19.981490843s) [seg_pos={0 0}, upload=472203]: Begin upload segment... Dec 28 01:55:20.110461169 (538h26m19.981544204s) [seg_pos={0 0}, upload=472203]: Done begin upload segment. Dec 28 01:55:20.110508519 (538h26m19.981591564s) [noise=true , address=145.79.227.36:4517 , piece_id=4HZU2GVP42BUXOC7I4BPGKMWDTY62GBF2U62T7WZIM25RY4622JA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.111363868 (538h26m19.982446903s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.111852699 (538h26m19.982935734s) [noise=true , address=45.91.116.5:4297 , piece_id=DJYTQCTMOUAJNCYXNEPUBBS7NC6L4CEPN2O5PZC2ZXOGAYBJ7KEQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.112289511 (538h26m19.983372546s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.112734107 (538h26m19.983817152s) [noise=true , address=145.79.227.11:4731 , piece_id=VCTCPEGZGZ36LRKYWJGBFZFAENPNQRKNU6CR47W2FH6S56DSRAUA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.113170789 (538h26m19.984253824s) [noise=true , address=145.79.227.46:4602 , piece_id=GDZWPYZFGJEO5QSP2GYVCWA7M2XFKVHSAJ7V2V2NWIAPI42CBPQQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.113535933 (538h26m19.984618968s) [noise=true , address=145.79.227.19:4077 , piece_id=VIVYUOXLZ3LMPSKWGMOSETNGJWCKFEEUR3V5COIJBJ34RG5G6OVQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.115039998 (538h26m19.986123033s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.116762938 (538h26m19.987845984s) [noise=true , address=45.91.116.6:4976 , piece_id=JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.119031007 (538h26m19.990114042s) [noise=true , address=45.91.116.8:4047 , piece_id=AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.120729502 (538h26m19.991812537s) [noise=true , address=145.79.227.9:4367 , piece_id=KHAKD3FVZGBMG23B7AHJNCHGCN4BCO2JCYPM3LDZB75K72NIB64A, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.121030786 (538h26m19.992113821s) [noise=true , address=145.79.227.3:4150 , piece_id=A4PNA3F62NEBXREBHY7WECL4U7CAKLQMJDJOWTRNUUZPN5UGJXLQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.122758064 (538h26m19.993841109s) [noise=true , address=145.79.227.51:4411 , piece_id=V2RTOKKPADMFYFDKQU5NNH2VY6O475A45PGPDZEKBBHHES6MA7OA, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.124502676 (538h26m19.995585721s) [noise=true , address=145.79.227.15:4874 , piece_id=E7OWZJ3XIK2S5H7ZKNRROSIPYRUA4ULA4SG6RZPLXMYDUV7NLIPQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.124782760 (538h26m19.995865795s) [noise=true , address=145.79.227.31:4798 , piece_id=TRP43YNIWDSRTICXPRUEPMH2CFMXRHCVMAGGKHU7BNXYCFASKKTQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:20.127995918 (538h26m19.999078963s) [noise=true , address=45.91.116.14:4191 , piece_id=V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ, seg_pos={0 0}, upload=472203]: Uploading piece... Dec 28 01:55:21.841594073 (538h26m21.712677108s) [upload=472203]: Next returned nil segment Dec 28 01:55:21.841607999 (538h26m21.712691034s) [upload=472203]: Waiting for error group managing segments... Dec 28 01:55:22.131885026 (538h26m22.002968061s) [noise=true , address=145.79.227.51:4411 , piece_id=V2RTOKKPADMFYFDKQU5NNH2VY6O475A45PGPDZEKBBHHES6MA7OA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.132670642 (538h26m22.003753677s) [noise=true , address=145.79.227.3:4150 , piece_id=A4PNA3F62NEBXREBHY7WECL4U7CAKLQMJDJOWTRNUUZPN5UGJXLQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.135529235 (538h26m22.00661227s) [noise=true , address=145.79.227.31:4798 , piece_id=TRP43YNIWDSRTICXPRUEPMH2CFMXRHCVMAGGKHU7BNXYCFASKKTQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.143997921 (538h26m22.015080966s) [noise=true , address=145.79.227.46:4602 , piece_id=GDZWPYZFGJEO5QSP2GYVCWA7M2XFKVHSAJ7V2V2NWIAPI42CBPQQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.171746960 (538h26m22.042829995s) [noise=true , address=145.79.227.36:4517 , piece_id=4HZU2GVP42BUXOC7I4BPGKMWDTY62GBF2U62T7WZIM25RY4622JA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.177885619 (538h26m22.048968655s) [noise=true , address=145.79.227.9:4367 , piece_id=KHAKD3FVZGBMG23B7AHJNCHGCN4BCO2JCYPM3LDZB75K72NIB64A, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.192891123 (538h26m22.063974158s) [noise=true , address=145.79.227.11:4731 , piece_id=VCTCPEGZGZ36LRKYWJGBFZFAENPNQRKNU6CR47W2FH6S56DSRAUA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.195234274 (538h26m22.066317309s) [noise=true , address=145.79.227.19:4077 , piece_id=VIVYUOXLZ3LMPSKWGMOSETNGJWCKFEEUR3V5COIJBJ34RG5G6OVQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.214051786 (538h26m22.085134821s) [noise=true , address=145.79.227.15:4874 , piece_id=E7OWZJ3XIK2S5H7ZKNRROSIPYRUA4ULA4SG6RZPLXMYDUV7NLIPQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.364054808 (538h26m22.235137843s) [noise=true , address=45.91.116.8:4047 , piece_id=AMN6KLGAIWHQ53YT4LJUN5BH2QWLIOCZMVAOCUARCDBWGJNQM3HA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.399898760 (538h26m22.270981795s) [noise=true , address=45.91.116.14:4191 , piece_id=V3YMLD42ZZUMH2KADHFTSOZJ6GLJTERJR4Z42P6JVZDQ2VO5I2YQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.419378313 (538h26m22.290461348s) [noise=true , address=45.91.116.6:4976 , piece_id=JW5LLTLKWCQOZSOTVYQKY3FGCGK6AOZ3KDQTD4TQQNTM7LDUKV4A, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.511193444 (538h26m22.382276479s) [noise=true , address=45.91.116.5:4297 , piece_id=DJYTQCTMOUAJNCYXNEPUBBS7NC6L4CEPN2O5PZC2ZXOGAYBJ7KEQ, seg_pos={0 0}, upload=472203]: Done uploading piece. err: <nil> Dec 28 01:55:22.511214024 (538h26m22.382297059s) [seg_pos={0 0}, upload=472203]: Segment reached optimal threshold of 13 pieces. Dec 28 01:55:22.511382495 (538h26m22.38246553s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35386->145.79.227.7:4295: use of closed network connection Dec 28 01:55:22.511387975 (538h26m22.38247101s) [noise=true , address=145.79.227.7:4295 , piece_id=7Z3JMWWEG4SCI46BTWL5V6G6XHJBIRBTXQTCSZZYTMBVSHZ6BCRA, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511578598 (538h26m22.382661633s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 01:55:22.511584520 (538h26m22.382667565s) [noise=true , address=145.79.227.33:4869 , piece_id=T5HTYSHITIL2DS4P3UN6HX3HTNJ7ZC65F5DTTLQXTZUZBRMNXF4Q, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511721460 (538h26m22.382804495s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:55:22.511754663 (538h26m22.382837718s) [noise=true , address=45.91.116.4:4939 , piece_id=NKFFUBPFSSJXS2KT6JMJSJJFN6PM7ODJNORXKGEEBWQDGJATXMOA, seg_pos={0 0}, upload=472203]: Long tail cancellation due to optimal threshold Dec 28 01:55:22.511787535 (538h26m22.382870571s) [seg_pos={0 0}, upload=472203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:55:22.511947380 (538h26m22.383030425s) [upload=472203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:55:22.511955776 (538h26m22.383038821s) [upload=472203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:55:22.829773646 (538h26m22.700856691s) [upload=472203]: Done uploading segments... Dec 28 01:55:22.829829423 (538h26m22.700912488s) [upload=472203]: Upload finished. err: <nil> Duration: 30.003207987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.] duration: 12.610610238s trace: https://jaeger.dev.storj.io/trace/ddf638e3e88995d FAIL (2025-12-28 01:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA) context deadline exceeded ======LOGS====== Dec 28 01:53:51.051310438 (538h24m50.922393474s) [upload=472184]: Starting upload Dec 28 01:53:51.051365514 (538h24m50.922448559s) [upload=472184]: Done starting upload Dec 28 01:53:51.051569432 (538h24m50.922652487s) [upload=472184]: Uploading segments... Dec 28 01:53:51.051679942 (538h24m50.922762977s) [upload=472184]: Got next segment. Inline: false Dec 28 01:53:51.051727653 (538h24m50.922810698s) [upload=472184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:53:51.051735989 (538h24m50.922819024s) [upload=472184]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:53:51.299314070 (538h24m51.170397105s) [seg_pos={0 0}, upload=472184]: Begin upload segment... Dec 28 01:53:51.299371058 (538h24m51.170454103s) [seg_pos={0 0}, upload=472184]: Done begin upload segment. Dec 28 01:53:51.299432976 (538h24m51.170516011s) [noise=true , address=145.79.227.9:4367 , piece_id=6QGO7Y77ICH6Y3RV7QKS6HE2JCHRGAGAFXYCQUPI5OUTQHE75B5A, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.299557584 (538h24m51.170640629s) [noise=true , address=145.79.227.11:4729 , piece_id=SMMSQCP24UMII343RL3XATTYU745LPHT5TF7IKRIBCRSSEVHJY2Q, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.299971081 (538h24m51.171054116s) [noise=true , address=145.79.227.13:4803 , piece_id=TTBMX4H7KHFLYWY5GJS52F45TKTGNRAWDXLCZXALIWBHRVGNEVSQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.300437449 (538h24m51.171520504s) [noise=true , address=145.79.227.12:4767 , piece_id=VKGXNUT7CXTZTPS5MHSYRQ5IM23MCAARBJ5QCHYXHHAFXXV2FZDA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.300888277 (538h24m51.171971322s) [noise=true , address=145.79.227.3:4151 , piece_id=BSN5AV2W66Z4MNHATEAI5364F662WSPZA6L7QD2BHOLGEKJHSUCA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301241550 (538h24m51.172324585s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301527234 (538h24m51.172610269s) [noise=true , address=145.79.227.52:4447 , piece_id=2IDTWPCNP7EL6MGJ5CJCZKYM32WWADW7ZPDG5HXQQVOLMZRRE3FA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301993923 (538h24m51.173076968s) [noise=true , address=145.79.227.5:4223 , piece_id=JTMVS4WCNFPY2GTJ7TE7MIEZIFJUWHOG3YBLGBXM5GNLZOP5DPEA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302079125 (538h24m51.17316216s) [noise=true , address=77.90.31.19:4844 , piece_id=FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302346073 (538h24m51.173429118s) [noise=true , address=45.91.116.13:4154 , piece_id=53FT5Z73CAOJ56FSHGMLVZIAGVVP5ADM7YXRUGAPI37RL4Q72RTA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302572444 (538h24m51.173655479s) [noise=true , address=145.79.227.36:4517 , piece_id=K763DK6T6E5YZIS3KWCP4BD6YZTQPDV356MPEDXOCFBLMILVWENA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302782083 (538h24m51.173865118s) [noise=true , address=145.79.227.14:4840 , piece_id=L4REI3TKATPMWEL2DPTOJVRRV2ICTH5357DG75HMWACXXXNKWJZQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303063780 (538h24m51.174146845s) [noise=true , address=45.91.116.7:4011 , piece_id=OGBBZ6RF2TCV77Z7XOGR2EX6WHRB4MDVVWBAAKJD7KF7ZNFIOAYA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303120588 (538h24m51.174203623s) [noise=true , address=145.79.227.19:4077 , piece_id=UJ62TTNU4NHPGTVBMK4SSD7GCZNRVVMDR6JJARE47J23VWFEHELA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303468781 (538h24m51.174551816s) [noise=true , address=45.91.116.11:4120 , piece_id=7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303827063 (538h24m51.174910108s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:52.908082601 (538h24m52.779165636s) [upload=472184]: Next returned nil segment Dec 28 01:53:52.908136745 (538h24m52.77921978s) [upload=472184]: Waiting for error group managing segments... Dec 28 01:53:53.303509582 (538h24m53.174592627s) [noise=true , address=145.79.227.11:4729 , piece_id=SMMSQCP24UMII343RL3XATTYU745LPHT5TF7IKRIBCRSSEVHJY2Q, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.304526889 (538h24m53.175609924s) [noise=true , address=145.79.227.52:4447 , piece_id=2IDTWPCNP7EL6MGJ5CJCZKYM32WWADW7ZPDG5HXQQVOLMZRRE3FA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.307026679 (538h24m53.178109724s) [noise=true , address=145.79.227.13:4803 , piece_id=TTBMX4H7KHFLYWY5GJS52F45TKTGNRAWDXLCZXALIWBHRVGNEVSQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.307296684 (538h24m53.178379719s) [noise=true , address=145.79.227.19:4077 , piece_id=UJ62TTNU4NHPGTVBMK4SSD7GCZNRVVMDR6JJARE47J23VWFEHELA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.308662915 (538h24m53.17974596s) [noise=true , address=145.79.227.5:4223 , piece_id=JTMVS4WCNFPY2GTJ7TE7MIEZIFJUWHOG3YBLGBXM5GNLZOP5DPEA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.311222169 (538h24m53.182305214s) [noise=true , address=145.79.227.36:4517 , piece_id=K763DK6T6E5YZIS3KWCP4BD6YZTQPDV356MPEDXOCFBLMILVWENA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.312173820 (538h24m53.183256855s) [noise=true , address=145.79.227.3:4151 , piece_id=BSN5AV2W66Z4MNHATEAI5364F662WSPZA6L7QD2BHOLGEKJHSUCA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.313367513 (538h24m53.184450558s) [noise=true , address=145.79.227.14:4840 , piece_id=L4REI3TKATPMWEL2DPTOJVRRV2ICTH5357DG75HMWACXXXNKWJZQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.319449134 (538h24m53.190532169s) [noise=true , address=145.79.227.9:4367 , piece_id=6QGO7Y77ICH6Y3RV7QKS6HE2JCHRGAGAFXYCQUPI5OUTQHE75B5A, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.321228021 (538h24m53.192311066s) [noise=true , address=145.79.227.12:4767 , piece_id=VKGXNUT7CXTZTPS5MHSYRQ5IM23MCAARBJ5QCHYXHHAFXXV2FZDA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.465953133 (538h24m53.337036168s) [noise=true , address=77.90.31.19:4844 , piece_id=FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.642141245 (538h24m53.51322428s) [noise=true , address=45.91.116.13:4154 , piece_id=53FT5Z73CAOJ56FSHGMLVZIAGVVP5ADM7YXRUGAPI37RL4Q72RTA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.661058396 (538h24m53.532141452s) [noise=true , address=45.91.116.7:4011 , piece_id=OGBBZ6RF2TCV77Z7XOGR2EX6WHRB4MDVVWBAAKJD7KF7ZNFIOAYA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.661080499 (538h24m53.532163564s) [seg_pos={0 0}, upload=472184]: Segment reached optimal threshold of 13 pieces. Dec 28 01:53:53.661334613 (538h24m53.532417648s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection Dec 28 01:53:53.661342738 (538h24m53.532425783s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Long tail cancellation due to optimal threshold Dec 28 01:53:53.661940787 (538h24m53.533023832s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection Dec 28 01:53:53.661951527 (538h24m53.533034573s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Long tail cancellation due to optimal threshold Dec 28 01:53:53.662272709 (538h24m53.533355744s) [noise=true , address=45.91.116.11:4120 , piece_id=7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.662298588 (538h24m53.533381633s) [seg_pos={0 0}, upload=472184]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 01:53:53.662391595 (538h24m53.53347464s) [upload=472184]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:53:53.662398178 (538h24m53.533481223s) [upload=472184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:53:53.963844523 (538h24m53.834927569s) [upload=472184]: Done uploading segments... Dec 28 01:53:53.963871594 (538h24m53.834954639s) [upload=472184]: Upload finished. err: <nil> Duration: 12.610640256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X] duration: 30.003400334s trace: https://jaeger.dev.storj.io/trace/ddf638e3e88995d FAIL (2025-12-28 01:54:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA) context deadline exceeded ======LOGS====== Dec 28 01:53:51.051310438 (538h24m50.922393474s) [upload=472184]: Starting upload Dec 28 01:53:51.051365514 (538h24m50.922448559s) [upload=472184]: Done starting upload Dec 28 01:53:51.051569432 (538h24m50.922652487s) [upload=472184]: Uploading segments... Dec 28 01:53:51.051679942 (538h24m50.922762977s) [upload=472184]: Got next segment. Inline: false Dec 28 01:53:51.051727653 (538h24m50.922810698s) [upload=472184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:53:51.051735989 (538h24m50.922819024s) [upload=472184]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:53:51.299314070 (538h24m51.170397105s) [seg_pos={0 0}, upload=472184]: Begin upload segment... Dec 28 01:53:51.299371058 (538h24m51.170454103s) [seg_pos={0 0}, upload=472184]: Done begin upload segment. Dec 28 01:53:51.299432976 (538h24m51.170516011s) [noise=true , address=145.79.227.9:4367 , piece_id=6QGO7Y77ICH6Y3RV7QKS6HE2JCHRGAGAFXYCQUPI5OUTQHE75B5A, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.299557584 (538h24m51.170640629s) [noise=true , address=145.79.227.11:4729 , piece_id=SMMSQCP24UMII343RL3XATTYU745LPHT5TF7IKRIBCRSSEVHJY2Q, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.299971081 (538h24m51.171054116s) [noise=true , address=145.79.227.13:4803 , piece_id=TTBMX4H7KHFLYWY5GJS52F45TKTGNRAWDXLCZXALIWBHRVGNEVSQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.300437449 (538h24m51.171520504s) [noise=true , address=145.79.227.12:4767 , piece_id=VKGXNUT7CXTZTPS5MHSYRQ5IM23MCAARBJ5QCHYXHHAFXXV2FZDA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.300888277 (538h24m51.171971322s) [noise=true , address=145.79.227.3:4151 , piece_id=BSN5AV2W66Z4MNHATEAI5364F662WSPZA6L7QD2BHOLGEKJHSUCA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301241550 (538h24m51.172324585s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301527234 (538h24m51.172610269s) [noise=true , address=145.79.227.52:4447 , piece_id=2IDTWPCNP7EL6MGJ5CJCZKYM32WWADW7ZPDG5HXQQVOLMZRRE3FA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.301993923 (538h24m51.173076968s) [noise=true , address=145.79.227.5:4223 , piece_id=JTMVS4WCNFPY2GTJ7TE7MIEZIFJUWHOG3YBLGBXM5GNLZOP5DPEA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302079125 (538h24m51.17316216s) [noise=true , address=77.90.31.19:4844 , piece_id=FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302346073 (538h24m51.173429118s) [noise=true , address=45.91.116.13:4154 , piece_id=53FT5Z73CAOJ56FSHGMLVZIAGVVP5ADM7YXRUGAPI37RL4Q72RTA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302572444 (538h24m51.173655479s) [noise=true , address=145.79.227.36:4517 , piece_id=K763DK6T6E5YZIS3KWCP4BD6YZTQPDV356MPEDXOCFBLMILVWENA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.302782083 (538h24m51.173865118s) [noise=true , address=145.79.227.14:4840 , piece_id=L4REI3TKATPMWEL2DPTOJVRRV2ICTH5357DG75HMWACXXXNKWJZQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303063780 (538h24m51.174146845s) [noise=true , address=45.91.116.7:4011 , piece_id=OGBBZ6RF2TCV77Z7XOGR2EX6WHRB4MDVVWBAAKJD7KF7ZNFIOAYA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303120588 (538h24m51.174203623s) [noise=true , address=145.79.227.19:4077 , piece_id=UJ62TTNU4NHPGTVBMK4SSD7GCZNRVVMDR6JJARE47J23VWFEHELA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303468781 (538h24m51.174551816s) [noise=true , address=45.91.116.11:4120 , piece_id=7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:51.303827063 (538h24m51.174910108s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Uploading piece... Dec 28 01:53:52.908082601 (538h24m52.779165636s) [upload=472184]: Next returned nil segment Dec 28 01:53:52.908136745 (538h24m52.77921978s) [upload=472184]: Waiting for error group managing segments... Dec 28 01:53:53.303509582 (538h24m53.174592627s) [noise=true , address=145.79.227.11:4729 , piece_id=SMMSQCP24UMII343RL3XATTYU745LPHT5TF7IKRIBCRSSEVHJY2Q, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.304526889 (538h24m53.175609924s) [noise=true , address=145.79.227.52:4447 , piece_id=2IDTWPCNP7EL6MGJ5CJCZKYM32WWADW7ZPDG5HXQQVOLMZRRE3FA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.307026679 (538h24m53.178109724s) [noise=true , address=145.79.227.13:4803 , piece_id=TTBMX4H7KHFLYWY5GJS52F45TKTGNRAWDXLCZXALIWBHRVGNEVSQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.307296684 (538h24m53.178379719s) [noise=true , address=145.79.227.19:4077 , piece_id=UJ62TTNU4NHPGTVBMK4SSD7GCZNRVVMDR6JJARE47J23VWFEHELA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.308662915 (538h24m53.17974596s) [noise=true , address=145.79.227.5:4223 , piece_id=JTMVS4WCNFPY2GTJ7TE7MIEZIFJUWHOG3YBLGBXM5GNLZOP5DPEA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.311222169 (538h24m53.182305214s) [noise=true , address=145.79.227.36:4517 , piece_id=K763DK6T6E5YZIS3KWCP4BD6YZTQPDV356MPEDXOCFBLMILVWENA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.312173820 (538h24m53.183256855s) [noise=true , address=145.79.227.3:4151 , piece_id=BSN5AV2W66Z4MNHATEAI5364F662WSPZA6L7QD2BHOLGEKJHSUCA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.313367513 (538h24m53.184450558s) [noise=true , address=145.79.227.14:4840 , piece_id=L4REI3TKATPMWEL2DPTOJVRRV2ICTH5357DG75HMWACXXXNKWJZQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.319449134 (538h24m53.190532169s) [noise=true , address=145.79.227.9:4367 , piece_id=6QGO7Y77ICH6Y3RV7QKS6HE2JCHRGAGAFXYCQUPI5OUTQHE75B5A, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.321228021 (538h24m53.192311066s) [noise=true , address=145.79.227.12:4767 , piece_id=VKGXNUT7CXTZTPS5MHSYRQ5IM23MCAARBJ5QCHYXHHAFXXV2FZDA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.465953133 (538h24m53.337036168s) [noise=true , address=77.90.31.19:4844 , piece_id=FCHHWURN5GXHQTL3OJZ7OTGP7SSPZOO64PGZPS5IMD7WZ5E5TBGA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.642141245 (538h24m53.51322428s) [noise=true , address=45.91.116.13:4154 , piece_id=53FT5Z73CAOJ56FSHGMLVZIAGVVP5ADM7YXRUGAPI37RL4Q72RTA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.661058396 (538h24m53.532141452s) [noise=true , address=45.91.116.7:4011 , piece_id=OGBBZ6RF2TCV77Z7XOGR2EX6WHRB4MDVVWBAAKJD7KF7ZNFIOAYA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.661080499 (538h24m53.532163564s) [seg_pos={0 0}, upload=472184]: Segment reached optimal threshold of 13 pieces. Dec 28 01:53:53.661334613 (538h24m53.532417648s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50206->45.91.116.14:4190: use of closed network connection Dec 28 01:53:53.661342738 (538h24m53.532425783s) [noise=true , address=45.91.116.14:4190 , piece_id=YHNPDKWL6GWK3FWZFFT6T3FMWXBIE5FWABPGKIPIG3BWNDHNWNZA, seg_pos={0 0}, upload=472184]: Long tail cancellation due to optimal threshold Dec 28 01:53:53.661940787 (538h24m53.533023832s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59416->145.79.227.7:4293: use of closed network connection Dec 28 01:53:53.661951527 (538h24m53.533034573s) [noise=true , address=145.79.227.7:4293 , piece_id=TILTQ3JKPBJ6DPAPCPE4LYRHVVVAZV74RW2SL6XVYOXGKYKSTJDQ, seg_pos={0 0}, upload=472184]: Long tail cancellation due to optimal threshold Dec 28 01:53:53.662272709 (538h24m53.533355744s) [noise=true , address=45.91.116.11:4120 , piece_id=7NGIWKJT5ZPVDUU7RK6YI33QKPDZVGANRSDQMRIIYYOSZJ2SQ2BA, seg_pos={0 0}, upload=472184]: Done uploading piece. err: <nil> Dec 28 01:53:53.662298588 (538h24m53.533381633s) [seg_pos={0 0}, upload=472184]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 01:53:53.662391595 (538h24m53.53347464s) [upload=472184]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:53:53.662398178 (538h24m53.533481223s) [upload=472184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:53:53.963844523 (538h24m53.834927569s) [upload=472184]: Done uploading segments... Dec 28 01:53:53.963871594 (538h24m53.834954639s) [upload=472184]: Upload finished. err: <nil> Duration: 30.003425832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..] duration: 7.963613026s trace: https://jaeger.dev.storj.io/trace/9ce0aa4755c24e5 FAIL (2025-12-28 01:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:52:11.195646003 (538h23m11.066729048s) [upload=472160]: Starting upload Dec 28 01:52:11.195732808 (538h23m11.066815863s) [upload=472160]: Done starting upload Dec 28 01:52:11.195845072 (538h23m11.066928117s) [upload=472160]: Uploading segments... Dec 28 01:52:11.195984899 (538h23m11.067067944s) [upload=472160]: Got next segment. Inline: false Dec 28 01:52:11.196025345 (538h23m11.067108391s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:52:11.196030956 (538h23m11.067114001s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:52:11.439281748 (538h23m11.310364813s) [seg_pos={0 0}, upload=472160]: Begin upload segment... Dec 28 01:52:11.439338887 (538h23m11.310421922s) [seg_pos={0 0}, upload=472160]: Done begin upload segment. Dec 28 01:52:11.439373282 (538h23m11.310456317s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.439519120 (538h23m11.310602165s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440168166 (538h23m11.311251201s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440184837 (538h23m11.311267872s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440582926 (538h23m11.311665971s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440615898 (538h23m11.311698943s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440957458 (538h23m11.312040503s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441038864 (538h23m11.312121909s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441331741 (538h23m11.312414776s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442099453 (538h23m11.313182498s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442309373 (538h23m11.313392418s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443040956 (538h23m11.314124001s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443477347 (538h23m11.314560392s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.444150268 (538h23m11.315233313s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.445281944 (538h23m11.316364989s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.448344586 (538h23m11.319427621s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:13.167805094 (538h23m13.038888129s) [upload=472160]: Next returned nil segment Dec 28 01:52:13.167822778 (538h23m13.038905813s) [upload=472160]: Waiting for error group managing segments... Dec 28 01:52:13.476555557 (538h23m13.347638592s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.483409158 (538h23m13.354492193s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.668452012 (538h23m13.539535057s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.692014048 (538h23m13.563097083s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.694526111 (538h23m13.565609156s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.699995606 (538h23m13.571078651s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.738435853 (538h23m13.609518888s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.742756330 (538h23m13.613839375s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.874970408 (538h23m13.746053473s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.956023847 (538h23m13.827106892s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.201790880 (538h23m14.072873925s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.226569162 (538h23m14.097652207s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703495792 (538h23m14.574578837s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703514318 (538h23m14.574597353s) [seg_pos={0 0}, upload=472160]: Segment reached optimal threshold of 13 pieces. Dec 28 01:52:14.703774553 (538h23m14.574857588s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703782419 (538h23m14.574865464s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.703956611 (538h23m14.575039646s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703962181 (538h23m14.575045216s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704143907 (538h23m14.575226942s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.704149638 (538h23m14.575232673s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704165058 (538h23m14.575248093s) [seg_pos={0 0}, upload=472160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:52:14.704231273 (538h23m14.575314318s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:52:14.704236844 (538h23m14.575319879s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:52:15.028165660 (538h23m14.899248705s) [upload=472160]: Done uploading segments... Dec 28 01:52:15.028184856 (538h23m14.899267901s) [upload=472160]: Upload finished. err: <nil> Duration: 7.963643263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X.] duration: 7.330343983s trace: https://jaeger.dev.storj.io/trace/9ce0aa4755c24e5 FAIL (2025-12-28 01:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:52:11.195646003 (538h23m11.066729048s) [upload=472160]: Starting upload Dec 28 01:52:11.195732808 (538h23m11.066815863s) [upload=472160]: Done starting upload Dec 28 01:52:11.195845072 (538h23m11.066928117s) [upload=472160]: Uploading segments... Dec 28 01:52:11.195984899 (538h23m11.067067944s) [upload=472160]: Got next segment. Inline: false Dec 28 01:52:11.196025345 (538h23m11.067108391s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:52:11.196030956 (538h23m11.067114001s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:52:11.439281748 (538h23m11.310364813s) [seg_pos={0 0}, upload=472160]: Begin upload segment... Dec 28 01:52:11.439338887 (538h23m11.310421922s) [seg_pos={0 0}, upload=472160]: Done begin upload segment. Dec 28 01:52:11.439373282 (538h23m11.310456317s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.439519120 (538h23m11.310602165s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440168166 (538h23m11.311251201s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440184837 (538h23m11.311267872s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440582926 (538h23m11.311665971s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440615898 (538h23m11.311698943s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440957458 (538h23m11.312040503s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441038864 (538h23m11.312121909s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441331741 (538h23m11.312414776s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442099453 (538h23m11.313182498s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442309373 (538h23m11.313392418s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443040956 (538h23m11.314124001s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443477347 (538h23m11.314560392s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.444150268 (538h23m11.315233313s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.445281944 (538h23m11.316364989s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.448344586 (538h23m11.319427621s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:13.167805094 (538h23m13.038888129s) [upload=472160]: Next returned nil segment Dec 28 01:52:13.167822778 (538h23m13.038905813s) [upload=472160]: Waiting for error group managing segments... Dec 28 01:52:13.476555557 (538h23m13.347638592s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.483409158 (538h23m13.354492193s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.668452012 (538h23m13.539535057s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.692014048 (538h23m13.563097083s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.694526111 (538h23m13.565609156s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.699995606 (538h23m13.571078651s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.738435853 (538h23m13.609518888s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.742756330 (538h23m13.613839375s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.874970408 (538h23m13.746053473s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.956023847 (538h23m13.827106892s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.201790880 (538h23m14.072873925s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.226569162 (538h23m14.097652207s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703495792 (538h23m14.574578837s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703514318 (538h23m14.574597353s) [seg_pos={0 0}, upload=472160]: Segment reached optimal threshold of 13 pieces. Dec 28 01:52:14.703774553 (538h23m14.574857588s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703782419 (538h23m14.574865464s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.703956611 (538h23m14.575039646s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703962181 (538h23m14.575045216s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704143907 (538h23m14.575226942s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.704149638 (538h23m14.575232673s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704165058 (538h23m14.575248093s) [seg_pos={0 0}, upload=472160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:52:14.704231273 (538h23m14.575314318s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:52:14.704236844 (538h23m14.575319879s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:52:15.028165660 (538h23m14.899248705s) [upload=472160]: Done uploading segments... Dec 28 01:52:15.028184856 (538h23m14.899267901s) [upload=472160]: Upload finished. err: <nil> Duration: 7.330373809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X] duration: 30.002859414s trace: https://jaeger.dev.storj.io/trace/9ce0aa4755c24e5 FAIL (2025-12-28 01:52:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:52:11.195646003 (538h23m11.066729048s) [upload=472160]: Starting upload Dec 28 01:52:11.195732808 (538h23m11.066815863s) [upload=472160]: Done starting upload Dec 28 01:52:11.195845072 (538h23m11.066928117s) [upload=472160]: Uploading segments... Dec 28 01:52:11.195984899 (538h23m11.067067944s) [upload=472160]: Got next segment. Inline: false Dec 28 01:52:11.196025345 (538h23m11.067108391s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:52:11.196030956 (538h23m11.067114001s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:52:11.439281748 (538h23m11.310364813s) [seg_pos={0 0}, upload=472160]: Begin upload segment... Dec 28 01:52:11.439338887 (538h23m11.310421922s) [seg_pos={0 0}, upload=472160]: Done begin upload segment. Dec 28 01:52:11.439373282 (538h23m11.310456317s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.439519120 (538h23m11.310602165s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440168166 (538h23m11.311251201s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440184837 (538h23m11.311267872s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440582926 (538h23m11.311665971s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440615898 (538h23m11.311698943s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.440957458 (538h23m11.312040503s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441038864 (538h23m11.312121909s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.441331741 (538h23m11.312414776s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442099453 (538h23m11.313182498s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.442309373 (538h23m11.313392418s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443040956 (538h23m11.314124001s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.443477347 (538h23m11.314560392s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.444150268 (538h23m11.315233313s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.445281944 (538h23m11.316364989s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:11.448344586 (538h23m11.319427621s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Uploading piece... Dec 28 01:52:13.167805094 (538h23m13.038888129s) [upload=472160]: Next returned nil segment Dec 28 01:52:13.167822778 (538h23m13.038905813s) [upload=472160]: Waiting for error group managing segments... Dec 28 01:52:13.476555557 (538h23m13.347638592s) [noise=true , address=145.79.227.7:4295 , piece_id=4MJFQEZTQEBRCNCCGK5XS2SA5Q3JAESF7EKUFWOXR6J3GRPUUIGQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.483409158 (538h23m13.354492193s) [noise=true , address=145.79.227.45:4230 , piece_id=HHURMWZJ6GN4SU4UQRMJOJAPAIF2UDUUUSVTM4RDGOYVTYLYIUUQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.668452012 (538h23m13.539535057s) [noise=true , address=45.91.116.6:4973 , piece_id=KKMXZI7RRS7X6SSHGMLGKSTJSTZUKVLW7CPU4BPZPASQYJVD5RFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.692014048 (538h23m13.563097083s) [noise=true , address=45.91.116.13:4155 , piece_id=UQ66XVETH3WFOM3ECHNROOHSSRPAQ3VUAND3Y247AFNWBSVDCEYQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.694526111 (538h23m13.565609156s) [noise=true , address=45.91.116.4:4939 , piece_id=KCALN5OKQLQM3VU67D3JA6X7WTMVOU3HCGREJII34CLBKYENORFA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.699995606 (538h23m13.571078651s) [noise=true , address=77.90.31.13:4627 , piece_id=DEX62FDL62LUOSWNOMQTTJWPYGV37FKUBMAU43LBSWGTAMAVBUQQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.738435853 (538h23m13.609518888s) [noise=true , address=77.90.31.14:4662 , piece_id=CX2WBVBEO7DDKOXIJRXRK56HSXSJIISS3WNHPRSLSPFZNQVZDB4A, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.742756330 (538h23m13.613839375s) [noise=true , address=45.91.116.11:4119 , piece_id=ORGPRB7JC3JVLXGBE6EFIDBUANZLTAYOMKSHXGU6NYSNFOY6FSOQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.874970408 (538h23m13.746053473s) [noise=true , address=45.91.116.5:4297 , piece_id=T4SHQ44SCAU6XGBU7ITQDWVMBGJZOG264BETFEUMAT5ZNCXDSNHA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:13.956023847 (538h23m13.827106892s) [noise=true , address=77.90.31.17:4770 , piece_id=P3NWZA4BEQWTLOCBGPQCSRU7PRTAKDBNHWJU72JEKYTCOM34XAJQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.201790880 (538h23m14.072873925s) [noise=true , address=145.79.227.5:4221 , piece_id=Q2F6ACD7VEUVJNO5AXHVNI2OWP2R3B2EFZUTBTYGGN62EY5VQNXQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.226569162 (538h23m14.097652207s) [noise=true , address=145.79.227.4:4187 , piece_id=7SEHNT6KJZXN2G4UR3R375XV3QISTU5JUTA6UWMZ5ZXPZONNKYDA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703495792 (538h23m14.574578837s) [noise=true , address=45.91.116.8:4045 , piece_id=S7JNHUINBABY7ETJR2DMKQN77GPKZ7VA4NJPRARXQJFWWVWHNBSQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: <nil> Dec 28 01:52:14.703514318 (538h23m14.574597353s) [seg_pos={0 0}, upload=472160]: Segment reached optimal threshold of 13 pieces. Dec 28 01:52:14.703774553 (538h23m14.574857588s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703782419 (538h23m14.574865464s) [noise=true , address=145.79.227.34:4906 , piece_id=6HLHARRNSGNGSI7T7UGUYV5RUG3EAMJTQYB532ANKVKIEZJ2TCTQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.703956611 (538h23m14.575039646s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.703962181 (538h23m14.575045216s) [noise=true , address=145.79.227.43:4158 , piece_id=3HGY7LZN6NUX4J7GXZEEP66YL7XDOMNI24CJDUQLWNYLDDYMGUMA, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704143907 (538h23m14.575226942s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 01:52:14.704149638 (538h23m14.575232673s) [noise=true , address=145.79.227.35:4482 , piece_id=3QSITWSPW6RSY52VW6PFPVKMFTETLF5GPKLTB3VBN7RJQAEKS4FQ, seg_pos={0 0}, upload=472160]: Long tail cancellation due to optimal threshold Dec 28 01:52:14.704165058 (538h23m14.575248093s) [seg_pos={0 0}, upload=472160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:52:14.704231273 (538h23m14.575314318s) [upload=472160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:52:14.704236844 (538h23m14.575319879s) [upload=472160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:52:15.028165660 (538h23m14.899248705s) [upload=472160]: Done uploading segments... Dec 28 01:52:15.028184856 (538h23m14.899267901s) [upload=472160]: Upload finished. err: <nil> Duration: 30.002890904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.] duration: 11.891780069s trace: https://jaeger.dev.storj.io/trace/1639e883a24123f1 FAIL (2025-12-28 01:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=472136]: Done starting upload Dec 28 01:50:30.611526120 (538h21m30.482609165s) [upload=472136]: Uploading segments... Dec 28 01:50:30.611814580 (538h21m30.482897625s) [upload=472136]: Got next segment. Inline: false Dec 28 01:50:30.611850919 (538h21m30.482933954s) [upload=472136]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:50:30.611855808 (538h21m30.482938853s) [upload=472136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:50:30.860124373 (538h21m30.731207418s) [seg_pos={0 0}, upload=472136]: Begin upload segment... Dec 28 01:50:30.860172355 (538h21m30.73125541s) [seg_pos={0 0}, upload=472136]: Done begin upload segment. Dec 28 01:50:30.860201691 (538h21m30.731284726s) [noise=true , address=145.79.227.13:4803 , piece_id=SCJPMMNDQZZ6A72X74M7BM6TX3YB3RC7K3W6OAXFDL3ULSATFEMQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.860232649 (538h21m30.731315684s) [noise=true , address=145.79.227.33:4870 , piece_id=CKWK3OSXCQ6LM27RLJPG7IXMV7CQJTQYY3U7UGFY43KVY3DJP6RA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.860815439 (538h21m30.731898485s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861220852 (538h21m30.732303897s) [noise=true , address=45.91.116.5:4300 , piece_id=YI6Q6WBAVTJRX5HFDBQACEDZLOC2ULTVVAPOWBIGE62TCBPO2HAA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861422175 (538h21m30.73250521s) [noise=true , address=45.91.116.12:4228 , piece_id=SP7Q66ZSTPZFXA7SCECPLXV3BQXA5T47TI3ULMZPZEHGWBPXX3SA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861651401 (538h21m30.732734436s) [noise=true , address=145.79.227.14:4838 , piece_id=VRNTPHEEO6X2R26ITGHBXO4NCFMPWDDBHR5IMBH4EN4CXFMFDJIQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861741563 (538h21m30.732824598s) [noise=true , address=145.79.227.3:4150 , piece_id=GBR7PPL552GKGV4Z7HC4V5NHLPSS2FVI7H2XZXI7O4EYRCL5BN5Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862061492 (538h21m30.733144527s) [noise=true , address=77.90.31.15:4700 , piece_id=RVH6ZTJWW3QP3NTZC5KW2CE7Y5ARZYL22DGG2KX7RQISHRDDEBQQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862066051 (538h21m30.733149086s) [noise=true , address=77.90.31.12:4590 , piece_id=MEJ4KAPEI3RGD2TLB2D4CUVRWEW733HILO6A4YHGWK2V4V4ZYH3Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862380089 (538h21m30.733463124s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862474088 (538h21m30.733557133s) [noise=true , address=45.91.116.9:4263 , piece_id=GGVN3ITK3PIFY6OHA24PBGRWQBLGUXLTAEDAHSHVPZSK34EQRNNQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862825137 (538h21m30.733908172s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863053070 (538h21m30.734136115s) [noise=true , address=45.91.116.6:4974 , piece_id=LYBLWAOUFSVPSHQM65DH7UM25UPZY7F2U37KPKEONPT24BVIWD4Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863189770 (538h21m30.734272805s) [noise=true , address=145.79.227.9:4367 , piece_id=JD65EXWMJMTUOJZUHMGR7ELOVL36HYL4R44AKF6FFILXVKJDC4HA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863416553 (538h21m30.734499588s) [noise=true , address=77.90.31.16:4736 , piece_id=FIVHCA7VNNA5LJOTW4MUH7JHGM2MDJQWLZAXSO7NYKUEUQRCSGOQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863509781 (538h21m30.734592816s) [noise=true , address=45.91.116.11:4119 , piece_id=F7ORE3ZKL5EVDPEYN26TBQI4QRL2HF4RUU7DBJPMD75CMNMWGR7Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:32.599894715 (538h21m32.470977761s) [upload=472136]: Next returned nil segment Dec 28 01:50:32.599910696 (538h21m32.470993742s) [upload=472136]: Waiting for error group managing segments... Dec 28 01:50:32.886979394 (538h21m32.758062439s) [noise=true , address=145.79.227.9:4367 , piece_id=JD65EXWMJMTUOJZUHMGR7ELOVL36HYL4R44AKF6FFILXVKJDC4HA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:32.901290174 (538h21m32.772373209s) [noise=true , address=145.79.227.14:4838 , piece_id=VRNTPHEEO6X2R26ITGHBXO4NCFMPWDDBHR5IMBH4EN4CXFMFDJIQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:32.961941775 (538h21m32.83302481s) [noise=true , address=145.79.227.13:4803 , piece_id=SCJPMMNDQZZ6A72X74M7BM6TX3YB3RC7K3W6OAXFDL3ULSATFEMQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.003066995 (538h21m32.87415004s) [noise=true , address=145.79.227.3:4150 , piece_id=GBR7PPL552GKGV4Z7HC4V5NHLPSS2FVI7H2XZXI7O4EYRCL5BN5Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.059832125 (538h21m32.93091516s) [noise=true , address=77.90.31.12:4590 , piece_id=MEJ4KAPEI3RGD2TLB2D4CUVRWEW733HILO6A4YHGWK2V4V4ZYH3Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.090280885 (538h21m32.96136392s) [noise=true , address=145.79.227.33:4870 , piece_id=CKWK3OSXCQ6LM27RLJPG7IXMV7CQJTQYY3U7UGFY43KVY3DJP6RA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.122515666 (538h21m32.993598711s) [noise=true , address=45.91.116.5:4300 , piece_id=YI6Q6WBAVTJRX5HFDBQACEDZLOC2ULTVVAPOWBIGE62TCBPO2HAA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.198644818 (538h21m33.069727863s) [noise=true , address=45.91.116.11:4119 , piece_id=F7ORE3ZKL5EVDPEYN26TBQI4QRL2HF4RUU7DBJPMD75CMNMWGR7Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.218243016 (538h21m33.089326051s) [noise=true , address=45.91.116.12:4228 , piece_id=SP7Q66ZSTPZFXA7SCECPLXV3BQXA5T47TI3ULMZPZEHGWBPXX3SA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.220606697 (538h21m33.091689752s) [noise=true , address=45.91.116.6:4974 , piece_id=LYBLWAOUFSVPSHQM65DH7UM25UPZY7F2U37KPKEONPT24BVIWD4Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.225751704 (538h21m33.096834749s) [noise=true , address=77.90.31.16:4736 , piece_id=FIVHCA7VNNA5LJOTW4MUH7JHGM2MDJQWLZAXSO7NYKUEUQRCSGOQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.279182796 (538h21m33.150265832s) [noise=true , address=77.90.31.15:4700 , piece_id=RVH6ZTJWW3QP3NTZC5KW2CE7Y5ARZYL22DGG2KX7RQISHRDDEBQQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.550065785 (538h21m33.42114883s) [noise=true , address=45.91.116.9:4263 , piece_id=GGVN3ITK3PIFY6OHA24PBGRWQBLGUXLTAEDAHSHVPZSK34EQRNNQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.550090551 (538h21m33.421173596s) [seg_pos={0 0}, upload=472136]: Segment reached optimal threshold of 13 pieces. Dec 28 01:50:33.550323686 (538h21m33.421406741s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 01:50:33.550334596 (538h21m33.421417631s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550393258 (538h21m33.421476293s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection Dec 28 01:50:33.550414870 (538h21m33.421497925s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550647612 (538h21m33.421730667s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:57618->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:57618->45.91.116.8:4045: use of closed network connection Dec 28 01:50:33.550657552 (538h21m33.421740617s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550741952 (538h21m33.421824997s) [seg_pos={0 0}, upload=472136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:50:33.550868974 (538h21m33.421952019s) [upload=472136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:50:33.550876739 (538h21m33.421959784s) [upload=472136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:50:33.840352070 (538h21m33.711435105s) [upload=472136]: Done uploading segments... Dec 28 01:50:33.840370875 (538h21m33.71145391s) [upload=472136]: Upload finished. err: <nil> Duration: 11.891797673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX] duration: 30.004006447s trace: https://jaeger.dev.storj.io/trace/1639e883a24123f1 FAIL (2025-12-28 01:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=472136]: Done starting upload Dec 28 01:50:30.611526120 (538h21m30.482609165s) [upload=472136]: Uploading segments... Dec 28 01:50:30.611814580 (538h21m30.482897625s) [upload=472136]: Got next segment. Inline: false Dec 28 01:50:30.611850919 (538h21m30.482933954s) [upload=472136]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:50:30.611855808 (538h21m30.482938853s) [upload=472136]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:50:30.860124373 (538h21m30.731207418s) [seg_pos={0 0}, upload=472136]: Begin upload segment... Dec 28 01:50:30.860172355 (538h21m30.73125541s) [seg_pos={0 0}, upload=472136]: Done begin upload segment. Dec 28 01:50:30.860201691 (538h21m30.731284726s) [noise=true , address=145.79.227.13:4803 , piece_id=SCJPMMNDQZZ6A72X74M7BM6TX3YB3RC7K3W6OAXFDL3ULSATFEMQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.860232649 (538h21m30.731315684s) [noise=true , address=145.79.227.33:4870 , piece_id=CKWK3OSXCQ6LM27RLJPG7IXMV7CQJTQYY3U7UGFY43KVY3DJP6RA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.860815439 (538h21m30.731898485s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861220852 (538h21m30.732303897s) [noise=true , address=45.91.116.5:4300 , piece_id=YI6Q6WBAVTJRX5HFDBQACEDZLOC2ULTVVAPOWBIGE62TCBPO2HAA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861422175 (538h21m30.73250521s) [noise=true , address=45.91.116.12:4228 , piece_id=SP7Q66ZSTPZFXA7SCECPLXV3BQXA5T47TI3ULMZPZEHGWBPXX3SA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861651401 (538h21m30.732734436s) [noise=true , address=145.79.227.14:4838 , piece_id=VRNTPHEEO6X2R26ITGHBXO4NCFMPWDDBHR5IMBH4EN4CXFMFDJIQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.861741563 (538h21m30.732824598s) [noise=true , address=145.79.227.3:4150 , piece_id=GBR7PPL552GKGV4Z7HC4V5NHLPSS2FVI7H2XZXI7O4EYRCL5BN5Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862061492 (538h21m30.733144527s) [noise=true , address=77.90.31.15:4700 , piece_id=RVH6ZTJWW3QP3NTZC5KW2CE7Y5ARZYL22DGG2KX7RQISHRDDEBQQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862066051 (538h21m30.733149086s) [noise=true , address=77.90.31.12:4590 , piece_id=MEJ4KAPEI3RGD2TLB2D4CUVRWEW733HILO6A4YHGWK2V4V4ZYH3Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862380089 (538h21m30.733463124s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862474088 (538h21m30.733557133s) [noise=true , address=45.91.116.9:4263 , piece_id=GGVN3ITK3PIFY6OHA24PBGRWQBLGUXLTAEDAHSHVPZSK34EQRNNQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.862825137 (538h21m30.733908172s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863053070 (538h21m30.734136115s) [noise=true , address=45.91.116.6:4974 , piece_id=LYBLWAOUFSVPSHQM65DH7UM25UPZY7F2U37KPKEONPT24BVIWD4Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863189770 (538h21m30.734272805s) [noise=true , address=145.79.227.9:4367 , piece_id=JD65EXWMJMTUOJZUHMGR7ELOVL36HYL4R44AKF6FFILXVKJDC4HA, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863416553 (538h21m30.734499588s) [noise=true , address=77.90.31.16:4736 , piece_id=FIVHCA7VNNA5LJOTW4MUH7JHGM2MDJQWLZAXSO7NYKUEUQRCSGOQ, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:30.863509781 (538h21m30.734592816s) [noise=true , address=45.91.116.11:4119 , piece_id=F7ORE3ZKL5EVDPEYN26TBQI4QRL2HF4RUU7DBJPMD75CMNMWGR7Q, seg_pos={0 0}, upload=472136]: Uploading piece... Dec 28 01:50:32.599894715 (538h21m32.470977761s) [upload=472136]: Next returned nil segment Dec 28 01:50:32.599910696 (538h21m32.470993742s) [upload=472136]: Waiting for error group managing segments... Dec 28 01:50:32.886979394 (538h21m32.758062439s) [noise=true , address=145.79.227.9:4367 , piece_id=JD65EXWMJMTUOJZUHMGR7ELOVL36HYL4R44AKF6FFILXVKJDC4HA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:32.901290174 (538h21m32.772373209s) [noise=true , address=145.79.227.14:4838 , piece_id=VRNTPHEEO6X2R26ITGHBXO4NCFMPWDDBHR5IMBH4EN4CXFMFDJIQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:32.961941775 (538h21m32.83302481s) [noise=true , address=145.79.227.13:4803 , piece_id=SCJPMMNDQZZ6A72X74M7BM6TX3YB3RC7K3W6OAXFDL3ULSATFEMQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.003066995 (538h21m32.87415004s) [noise=true , address=145.79.227.3:4150 , piece_id=GBR7PPL552GKGV4Z7HC4V5NHLPSS2FVI7H2XZXI7O4EYRCL5BN5Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.059832125 (538h21m32.93091516s) [noise=true , address=77.90.31.12:4590 , piece_id=MEJ4KAPEI3RGD2TLB2D4CUVRWEW733HILO6A4YHGWK2V4V4ZYH3Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.090280885 (538h21m32.96136392s) [noise=true , address=145.79.227.33:4870 , piece_id=CKWK3OSXCQ6LM27RLJPG7IXMV7CQJTQYY3U7UGFY43KVY3DJP6RA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.122515666 (538h21m32.993598711s) [noise=true , address=45.91.116.5:4300 , piece_id=YI6Q6WBAVTJRX5HFDBQACEDZLOC2ULTVVAPOWBIGE62TCBPO2HAA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.198644818 (538h21m33.069727863s) [noise=true , address=45.91.116.11:4119 , piece_id=F7ORE3ZKL5EVDPEYN26TBQI4QRL2HF4RUU7DBJPMD75CMNMWGR7Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.218243016 (538h21m33.089326051s) [noise=true , address=45.91.116.12:4228 , piece_id=SP7Q66ZSTPZFXA7SCECPLXV3BQXA5T47TI3ULMZPZEHGWBPXX3SA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.220606697 (538h21m33.091689752s) [noise=true , address=45.91.116.6:4974 , piece_id=LYBLWAOUFSVPSHQM65DH7UM25UPZY7F2U37KPKEONPT24BVIWD4Q, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.225751704 (538h21m33.096834749s) [noise=true , address=77.90.31.16:4736 , piece_id=FIVHCA7VNNA5LJOTW4MUH7JHGM2MDJQWLZAXSO7NYKUEUQRCSGOQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.279182796 (538h21m33.150265832s) [noise=true , address=77.90.31.15:4700 , piece_id=RVH6ZTJWW3QP3NTZC5KW2CE7Y5ARZYL22DGG2KX7RQISHRDDEBQQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.550065785 (538h21m33.42114883s) [noise=true , address=45.91.116.9:4263 , piece_id=GGVN3ITK3PIFY6OHA24PBGRWQBLGUXLTAEDAHSHVPZSK34EQRNNQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: <nil> Dec 28 01:50:33.550090551 (538h21m33.421173596s) [seg_pos={0 0}, upload=472136]: Segment reached optimal threshold of 13 pieces. Dec 28 01:50:33.550323686 (538h21m33.421406741s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 01:50:33.550334596 (538h21m33.421417631s) [noise=true , address=77.90.31.13:4629 , piece_id=FSYGPETBYTYWN3EMIUDJOYRKRZEX4LW52YAWPHYJMJPSVCY54AFA, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550393258 (538h21m33.421476293s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58562->145.79.227.37:4565: use of closed network connection Dec 28 01:50:33.550414870 (538h21m33.421497925s) [noise=true , address=145.79.227.37:4565 , piece_id=T6CGGJ5LXMYCNCNGMQF23WZGWU4LRSZAIOKGNNXMHLEQPIENZ4FQ, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550647612 (538h21m33.421730667s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:57618->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:57618->45.91.116.8:4045: use of closed network connection Dec 28 01:50:33.550657552 (538h21m33.421740617s) [noise=true , address=45.91.116.8:4045 , piece_id=5KLWTTH4L6EJJXJVN5AIHBGZVBJXAPOU3OVKSRHX4M6EATM7ZNOQ, seg_pos={0 0}, upload=472136]: Long tail cancellation due to optimal threshold Dec 28 01:50:33.550741952 (538h21m33.421824997s) [seg_pos={0 0}, upload=472136]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:50:33.550868974 (538h21m33.421952019s) [upload=472136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:50:33.550876739 (538h21m33.421959784s) [upload=472136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:50:33.840352070 (538h21m33.711435105s) [upload=472136]: Done uploading segments... Dec 28 01:50:33.840370875 (538h21m33.71145391s) [upload=472136]: Upload finished. err: <nil> Duration: 30.004031744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.X] duration: 30.002417541s trace: https://jaeger.dev.storj.io/trace/6f1595cb5de149a3 FAIL (2025-12-28 01:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:49:15.887499724 (538h20m15.758582759s) [upload=472121]: Starting upload Dec 28 01:49:15.887558366 (538h20m15.758641401s) [upload=472121]: Done starting upload Dec 28 01:49:15.887625604 (538h20m15.758708639s) [upload=472121]: Uploading segments... Dec 28 01:49:15.887739230 (538h20m15.758822265s) [upload=472121]: Got next segment. Inline: false Dec 28 01:49:15.887771923 (538h20m15.758854958s) [upload=472121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:49:15.887775810 (538h20m15.758858845s) [upload=472121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:49:16.133457821 (538h20m16.004540856s) [seg_pos={0 0}, upload=472121]: Begin upload segment... Dec 28 01:49:16.133497857 (538h20m16.004580892s) [seg_pos={0 0}, upload=472121]: Done begin upload segment. Dec 28 01:49:16.133521443 (538h20m16.004604478s) [noise=true , address=45.91.116.7:4010 , piece_id=U3YJPS3DF3WR7A7TGY4IL2XPBPNN5WZRNH5WPOEG3FFQ4HPXXKUQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.133559585 (538h20m16.00464262s) [noise=true , address=145.79.227.46:4601 , piece_id=HZK2BMABLHR6DGQBY7ZLCCVBGQEGRL3G3ZBM6RR3VZB6WKL7256A, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.134072863 (538h20m16.005155908s) [noise=true , address=145.79.227.12:4767 , piece_id=AYEAQVZF3GZZTYBATFP5CXEHZAQXA5JD43WZOGQJYPZDXUHZUKQQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.134500507 (538h20m16.005583552s) [noise=true , address=145.79.227.48:4303 , piece_id=2LMUG4376PZ7AJVIJYQ7YCR7ABZNHRXNJRLUMQXRHMXJUDMPKQMA, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.134928541 (538h20m16.006011577s) [noise=true , address=145.79.227.18:4006 , piece_id=FH7FR3SRHZS3R5RFYIWPRKVHF7GG4AUF55AIRKR3P2XY4MX7NVVQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.135294569 (538h20m16.006377604s) [noise=true , address=77.90.31.14:4662 , piece_id=CYVLLNTWVQXNYLANA3MJOCNV5BRY6NT7KC7RNQ5KP25Q2PT74ULQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.135637622 (538h20m16.006720657s) [noise=true , address=145.79.227.3:4151 , piece_id=JG6TGZRDEIAR43MFS5RDZDVCD5GDE5JCUX64SHSGVU4PIO5LB3GA, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.135971948 (538h20m16.007054983s) [noise=true , address=145.79.227.45:4231 , piece_id=NZDZB6HHN3DZJZRM7YXZOZECID5QPCKK4RUAKOB5CGMYT4MPYAYQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.136419891 (538h20m16.007502926s) [noise=true , address=145.79.227.47:4266 , piece_id=DCIKP4JURJOJ4IGGCNKEKADJBJMHHB3XYLBYI7BB3BOCYPKRP4IQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.137407112 (538h20m16.008490177s) [noise=true , address=145.79.227.17:4972 , piece_id=B2BTDJ6P7XLAZ5PBOYDGZEKOHLOMMOM73XTUKZY2DM4JKKAKM2TQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.138944599 (538h20m16.010027634s) [noise=true , address=145.79.227.44:4195 , piece_id=T235JT4OB5MRHHBOL6YAJEIALWS4U5PSFOUFFFD3BFVAZ5YKYUDA, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.138957854 (538h20m16.010040889s) [noise=true , address=45.91.116.9:4262 , piece_id=JP5KRTEXMFEW6M4DJCP6KL5K3IUCB2D5QGGSIWJGMHHEYZCYF4IA, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.140231971 (538h20m16.011315016s) [noise=true , address=145.79.227.52:4447 , piece_id=6RC5F4ZX4PJNYPV3OLUE6MWVNZUNT2WJRN5PRY7DVPOLUQED6MLQ, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.141432847 (538h20m16.012515892s) [noise=true , address=45.91.116.14:4190 , piece_id=E7HFKOIZMMQD7ANMUY2FM33EXTHILMTO2WKRM7KDYK45UJVOGD6A, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.142913447 (538h20m16.013996482s) [noise=true , address=45.91.116.13:4153 , piece_id=HTGPSKMBMJJDLZHML23P2RZ4SH77EPV3JNYSZISOXBJ32LT5SI7A, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:16.144497643 (538h20m16.015580678s) [noise=true , address=145.79.227.49:4337 , piece_id=UCANS5LKIPGBEUF3TU5TGV4OC7OV2K3SXJVAQ6ZPNX2GQFIDLKPA, seg_pos={0 0}, upload=472121]: Uploading piece... Dec 28 01:49:17.850482492 (538h20m17.721565547s) [upload=472121]: Next returned nil segment Dec 28 01:49:17.850510486 (538h20m17.721593541s) [upload=472121]: Waiting for error group managing segments... Dec 28 01:49:18.133327568 (538h20m18.004410613s) [noise=true , address=145.79.227.46:4601 , piece_id=HZK2BMABLHR6DGQBY7ZLCCVBGQEGRL3G3ZBM6RR3VZB6WKL7256A, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.140313089 (538h20m18.011396134s) [noise=true , address=145.79.227.48:4303 , piece_id=2LMUG4376PZ7AJVIJYQ7YCR7ABZNHRXNJRLUMQXRHMXJUDMPKQMA, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.144926965 (538h20m18.01601001s) [noise=true , address=145.79.227.17:4972 , piece_id=B2BTDJ6P7XLAZ5PBOYDGZEKOHLOMMOM73XTUKZY2DM4JKKAKM2TQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.146618866 (538h20m18.017701911s) [noise=true , address=145.79.227.12:4767 , piece_id=AYEAQVZF3GZZTYBATFP5CXEHZAQXA5JD43WZOGQJYPZDXUHZUKQQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.147181127 (538h20m18.018264162s) [noise=true , address=145.79.227.45:4231 , piece_id=NZDZB6HHN3DZJZRM7YXZOZECID5QPCKK4RUAKOB5CGMYT4MPYAYQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.149602798 (538h20m18.020685833s) [noise=true , address=145.79.227.18:4006 , piece_id=FH7FR3SRHZS3R5RFYIWPRKVHF7GG4AUF55AIRKR3P2XY4MX7NVVQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.166083250 (538h20m18.037166285s) [noise=true , address=145.79.227.3:4151 , piece_id=JG6TGZRDEIAR43MFS5RDZDVCD5GDE5JCUX64SHSGVU4PIO5LB3GA, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.174054268 (538h20m18.045137303s) [noise=true , address=145.79.227.47:4266 , piece_id=DCIKP4JURJOJ4IGGCNKEKADJBJMHHB3XYLBYI7BB3BOCYPKRP4IQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.174444321 (538h20m18.045527356s) [noise=true , address=145.79.227.52:4447 , piece_id=6RC5F4ZX4PJNYPV3OLUE6MWVNZUNT2WJRN5PRY7DVPOLUQED6MLQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.427572270 (538h20m18.298655305s) [noise=true , address=45.91.116.14:4190 , piece_id=E7HFKOIZMMQD7ANMUY2FM33EXTHILMTO2WKRM7KDYK45UJVOGD6A, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.441774022 (538h20m18.312857057s) [noise=true , address=77.90.31.14:4662 , piece_id=CYVLLNTWVQXNYLANA3MJOCNV5BRY6NT7KC7RNQ5KP25Q2PT74ULQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:18.567494545 (538h20m18.43857758s) [noise=true , address=45.91.116.9:4262 , piece_id=JP5KRTEXMFEW6M4DJCP6KL5K3IUCB2D5QGGSIWJGMHHEYZCYF4IA, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:19.062078068 (538h20m18.933161103s) [noise=true , address=45.91.116.7:4010 , piece_id=U3YJPS3DF3WR7A7TGY4IL2XPBPNN5WZRNH5WPOEG3FFQ4HPXXKUQ, seg_pos={0 0}, upload=472121]: Done uploading piece. err: <nil> Dec 28 01:49:19.062096112 (538h20m18.933179157s) [seg_pos={0 0}, upload=472121]: Segment reached optimal threshold of 13 pieces. Dec 28 01:49:19.062285212 (538h20m18.933368248s) [noise=true , address=45.91.116.13:4153 , piece_id=HTGPSKMBMJJDLZHML23P2RZ4SH77EPV3JNYSZISOXBJ32LT5SI7A, seg_pos={0 0}, upload=472121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:38772->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:38772->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38772->45.91.116.13:4153: use of closed network connection Dec 28 01:49:19.062292256 (538h20m18.933375291s) [noise=true , address=45.91.116.13:4153 , piece_id=HTGPSKMBMJJDLZHML23P2RZ4SH77EPV3JNYSZISOXBJ32LT5SI7A, seg_pos={0 0}, upload=472121]: Long tail cancellation due to optimal threshold Dec 28 01:49:19.062340347 (538h20m18.933423382s) [noise=true , address=145.79.227.49:4337 , piece_id=UCANS5LKIPGBEUF3TU5TGV4OC7OV2K3SXJVAQ6ZPNX2GQFIDLKPA, seg_pos={0 0}, upload=472121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 01:49:19.062348313 (538h20m18.933431348s) [noise=true , address=145.79.227.49:4337 , piece_id=UCANS5LKIPGBEUF3TU5TGV4OC7OV2K3SXJVAQ6ZPNX2GQFIDLKPA, seg_pos={0 0}, upload=472121]: Long tail cancellation due to optimal threshold Dec 28 01:49:19.062484451 (538h20m18.933567486s) [noise=true , address=145.79.227.44:4195 , piece_id=T235JT4OB5MRHHBOL6YAJEIALWS4U5PSFOUFFFD3BFVAZ5YKYUDA, seg_pos={0 0}, upload=472121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 01:49:19.062490053 (538h20m18.933573088s) [noise=true , address=145.79.227.44:4195 , piece_id=T235JT4OB5MRHHBOL6YAJEIALWS4U5PSFOUFFFD3BFVAZ5YKYUDA, seg_pos={0 0}, upload=472121]: Long tail cancellation due to optimal threshold Dec 28 01:49:19.062506363 (538h20m18.933589398s) [seg_pos={0 0}, upload=472121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:49:19.062605813 (538h20m18.933688848s) [upload=472121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:49:19.062614890 (538h20m18.933697925s) [upload=472121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:49:19.361648691 (538h20m19.232731727s) [upload=472121]: Done uploading segments... Dec 28 01:49:19.361692074 (538h20m19.232775109s) [upload=472121]: Upload finished. err: <nil> Duration: 30.002439111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.] duration: 6.341168054s trace: https://jaeger.dev.storj.io/trace/2cffb320711571f3 FAIL (2025-12-28 01:48:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 28 01:47:22.227469944 (538h18m22.098552979s) [upload=472098]: Got next segment. Inline: false Dec 28 01:47:22.227514228 (538h18m22.098597273s) [upload=472098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:47:22.227520671 (538h18m22.098603706s) [upload=472098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:47:22.470528990 (538h18m22.341612035s) [seg_pos={0 0}, upload=472098]: Begin upload segment... Dec 28 01:47:22.470570159 (538h18m22.341653194s) [seg_pos={0 0}, upload=472098]: Done begin upload segment. Dec 28 01:47:22.470598452 (538h18m22.341681487s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.470616246 (538h18m22.341699292s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.470734451 (538h18m22.341817486s) [noise=true , address=145.79.227.45:4230 , piece_id=4V2X6HF6UY2J6LUCWTQ4YMK4R3BEINILJZG2DJ4JNFVJQA4SN7TQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471046966 (538h18m22.342130001s) [noise=true , address=45.91.116.4:4937 , piece_id=VLLHKMFCHB5UNNMOJVZCOJVVL2Q2H7DXB3KX3DMDBJK7KBO7OQDQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471318353 (538h18m22.342401388s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVUEUNYY335H7SDW5IIBVRMQVUYP4AL32F35CSZHBDKRADPZBA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471631990 (538h18m22.342715025s) [noise=true , address=145.79.227.39:4015 , piece_id=7N723H5GPTQZDUP7SD3VKGWPAKJ5Y2YYMQV73FHOMFGD3KQPLF2A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471896314 (538h18m22.342979349s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.472239077 (538h18m22.343322112s) [noise=true , address=145.79.227.18:4006 , piece_id=PQXKMJALPMWQOXDENCWAH4Z42ODZIOQBPVH5ZJ2SJZIAC67HNRNA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.472544959 (538h18m22.343627994s) [noise=true , address=145.79.227.41:4087 , piece_id=LCQHAZEAM6YZL7UEV6BH2R6SAIL4KB2QPHZZ7RBSXYJA6RYV3IBQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.474215560 (538h18m22.345298605s) [noise=true , address=45.91.116.14:4191 , piece_id=JUOJBAQPTYDDVTHJZDBLQQSDOYIGFOGYVBWMYCOWCU3IWD2FV32A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.475382242 (538h18m22.346465277s) [noise=true , address=77.90.31.10:4518 , piece_id=IHGO3XDH2C6PCCWMGW2FFNSWSL45P4WFO4V755CLW2RI243FGCEQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.476367828 (538h18m22.347450863s) [noise=true , address=145.79.227.47:4267 , piece_id=VWDHNYPGB5ETN2Y444VEXM5R2BRCK63ASUZFHIP3SDTS2DBKVBCA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.477358365 (538h18m22.34844141s) [noise=true , address=77.90.31.12:4590 , piece_id=DMTHXF4ABIIJHDHLAYQBH3QIURA4HVVHV3G4AG6KL4KGCVOLVMKQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.478397182 (538h18m22.349480247s) [noise=true , address=45.91.116.5:4297 , piece_id=QAHSEFPCY3WFAZC7PVNH4XPQO46522XNNHP7RIWH4HBEMXP2CAYQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.479765248 (538h18m22.350848283s) [noise=true , address=145.79.227.7:4293 , piece_id=UQHJ4RVNOR7OUXKTAKRUZNBG7LKV2LXRWY7TJMBBKAXTDQ7EMP6A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.480969541 (538h18m22.352052576s) [noise=true , address=145.79.227.51:4409 , piece_id=5TADRJVLZWLTRXVEJCE247KFCVF6FDQBU6R4HEKJC35KPJPDSXBA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:24.142847111 (538h18m24.013930156s) [upload=472098]: Next returned nil segment Dec 28 01:47:24.142864334 (538h18m24.013947379s) [upload=472098]: Waiting for error group managing segments... Dec 28 01:47:24.469041922 (538h18m24.340124967s) [noise=true , address=145.79.227.18:4006 , piece_id=PQXKMJALPMWQOXDENCWAH4Z42ODZIOQBPVH5ZJ2SJZIAC67HNRNA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.471245709 (538h18m24.342328744s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVUEUNYY335H7SDW5IIBVRMQVUYP4AL32F35CSZHBDKRADPZBA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.471965839 (538h18m24.343048874s) [noise=true , address=145.79.227.39:4015 , piece_id=7N723H5GPTQZDUP7SD3VKGWPAKJ5Y2YYMQV73FHOMFGD3KQPLF2A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.477443911 (538h18m24.348526946s) [noise=true , address=145.79.227.7:4293 , piece_id=UQHJ4RVNOR7OUXKTAKRUZNBG7LKV2LXRWY7TJMBBKAXTDQ7EMP6A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.480718246 (538h18m24.351801281s) [noise=true , address=145.79.227.45:4230 , piece_id=4V2X6HF6UY2J6LUCWTQ4YMK4R3BEINILJZG2DJ4JNFVJQA4SN7TQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.482376363 (538h18m24.353459398s) [noise=true , address=145.79.227.41:4087 , piece_id=LCQHAZEAM6YZL7UEV6BH2R6SAIL4KB2QPHZZ7RBSXYJA6RYV3IBQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.495507397 (538h18m24.366590442s) [noise=true , address=145.79.227.51:4409 , piece_id=5TADRJVLZWLTRXVEJCE247KFCVF6FDQBU6R4HEKJC35KPJPDSXBA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.511878420 (538h18m24.382961455s) [noise=true , address=145.79.227.47:4267 , piece_id=VWDHNYPGB5ETN2Y444VEXM5R2BRCK63ASUZFHIP3SDTS2DBKVBCA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.745649270 (538h18m24.616732315s) [noise=true , address=77.90.31.12:4590 , piece_id=DMTHXF4ABIIJHDHLAYQBH3QIURA4HVVHV3G4AG6KL4KGCVOLVMKQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.754944830 (538h18m24.626027875s) [noise=true , address=45.91.116.5:4297 , piece_id=QAHSEFPCY3WFAZC7PVNH4XPQO46522XNNHP7RIWH4HBEMXP2CAYQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.763449264 (538h18m24.634532299s) [noise=true , address=77.90.31.10:4518 , piece_id=IHGO3XDH2C6PCCWMGW2FFNSWSL45P4WFO4V755CLW2RI243FGCEQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.771660770 (538h18m24.642743825s) [noise=true , address=45.91.116.4:4937 , piece_id=VLLHKMFCHB5UNNMOJVZCOJVVL2Q2H7DXB3KX3DMDBJK7KBO7OQDQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.788420704 (538h18m24.659503739s) [noise=true , address=45.91.116.14:4191 , piece_id=JUOJBAQPTYDDVTHJZDBLQQSDOYIGFOGYVBWMYCOWCU3IWD2FV32A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.788452504 (538h18m24.659535539s) [seg_pos={0 0}, upload=472098]: Segment reached optimal threshold of 13 pieces. Dec 28 01:47:24.788651243 (538h18m24.659734278s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection Dec 28 01:47:24.788657454 (538h18m24.659740489s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788840303 (538h18m24.659923348s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection Dec 28 01:47:24.788860261 (538h18m24.659943296s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788865641 (538h18m24.659948676s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:47:24.788872645 (538h18m24.65995568s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788898263 (538h18m24.659981298s) [seg_pos={0 0}, upload=472098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:47:24.788991481 (538h18m24.660074516s) [upload=472098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:47:24.788999416 (538h18m24.660082451s) [upload=472098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:47:25.074744052 (538h18m24.945827097s) [upload=472098]: Done uploading segments... Dec 28 01:47:25.074762378 (538h18m24.945845413s) [upload=472098]: Upload finished. err: <nil> Duration: 6.341198081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX] duration: 30.002510553s trace: https://jaeger.dev.storj.io/trace/2cffb320711571f3 FAIL (2025-12-28 01:47:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 28 01:47:22.227469944 (538h18m22.098552979s) [upload=472098]: Got next segment. Inline: false Dec 28 01:47:22.227514228 (538h18m22.098597273s) [upload=472098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:47:22.227520671 (538h18m22.098603706s) [upload=472098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:47:22.470528990 (538h18m22.341612035s) [seg_pos={0 0}, upload=472098]: Begin upload segment... Dec 28 01:47:22.470570159 (538h18m22.341653194s) [seg_pos={0 0}, upload=472098]: Done begin upload segment. Dec 28 01:47:22.470598452 (538h18m22.341681487s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.470616246 (538h18m22.341699292s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.470734451 (538h18m22.341817486s) [noise=true , address=145.79.227.45:4230 , piece_id=4V2X6HF6UY2J6LUCWTQ4YMK4R3BEINILJZG2DJ4JNFVJQA4SN7TQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471046966 (538h18m22.342130001s) [noise=true , address=45.91.116.4:4937 , piece_id=VLLHKMFCHB5UNNMOJVZCOJVVL2Q2H7DXB3KX3DMDBJK7KBO7OQDQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471318353 (538h18m22.342401388s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVUEUNYY335H7SDW5IIBVRMQVUYP4AL32F35CSZHBDKRADPZBA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471631990 (538h18m22.342715025s) [noise=true , address=145.79.227.39:4015 , piece_id=7N723H5GPTQZDUP7SD3VKGWPAKJ5Y2YYMQV73FHOMFGD3KQPLF2A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.471896314 (538h18m22.342979349s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.472239077 (538h18m22.343322112s) [noise=true , address=145.79.227.18:4006 , piece_id=PQXKMJALPMWQOXDENCWAH4Z42ODZIOQBPVH5ZJ2SJZIAC67HNRNA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.472544959 (538h18m22.343627994s) [noise=true , address=145.79.227.41:4087 , piece_id=LCQHAZEAM6YZL7UEV6BH2R6SAIL4KB2QPHZZ7RBSXYJA6RYV3IBQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.474215560 (538h18m22.345298605s) [noise=true , address=45.91.116.14:4191 , piece_id=JUOJBAQPTYDDVTHJZDBLQQSDOYIGFOGYVBWMYCOWCU3IWD2FV32A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.475382242 (538h18m22.346465277s) [noise=true , address=77.90.31.10:4518 , piece_id=IHGO3XDH2C6PCCWMGW2FFNSWSL45P4WFO4V755CLW2RI243FGCEQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.476367828 (538h18m22.347450863s) [noise=true , address=145.79.227.47:4267 , piece_id=VWDHNYPGB5ETN2Y444VEXM5R2BRCK63ASUZFHIP3SDTS2DBKVBCA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.477358365 (538h18m22.34844141s) [noise=true , address=77.90.31.12:4590 , piece_id=DMTHXF4ABIIJHDHLAYQBH3QIURA4HVVHV3G4AG6KL4KGCVOLVMKQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.478397182 (538h18m22.349480247s) [noise=true , address=45.91.116.5:4297 , piece_id=QAHSEFPCY3WFAZC7PVNH4XPQO46522XNNHP7RIWH4HBEMXP2CAYQ, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.479765248 (538h18m22.350848283s) [noise=true , address=145.79.227.7:4293 , piece_id=UQHJ4RVNOR7OUXKTAKRUZNBG7LKV2LXRWY7TJMBBKAXTDQ7EMP6A, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:22.480969541 (538h18m22.352052576s) [noise=true , address=145.79.227.51:4409 , piece_id=5TADRJVLZWLTRXVEJCE247KFCVF6FDQBU6R4HEKJC35KPJPDSXBA, seg_pos={0 0}, upload=472098]: Uploading piece... Dec 28 01:47:24.142847111 (538h18m24.013930156s) [upload=472098]: Next returned nil segment Dec 28 01:47:24.142864334 (538h18m24.013947379s) [upload=472098]: Waiting for error group managing segments... Dec 28 01:47:24.469041922 (538h18m24.340124967s) [noise=true , address=145.79.227.18:4006 , piece_id=PQXKMJALPMWQOXDENCWAH4Z42ODZIOQBPVH5ZJ2SJZIAC67HNRNA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.471245709 (538h18m24.342328744s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVUEUNYY335H7SDW5IIBVRMQVUYP4AL32F35CSZHBDKRADPZBA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.471965839 (538h18m24.343048874s) [noise=true , address=145.79.227.39:4015 , piece_id=7N723H5GPTQZDUP7SD3VKGWPAKJ5Y2YYMQV73FHOMFGD3KQPLF2A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.477443911 (538h18m24.348526946s) [noise=true , address=145.79.227.7:4293 , piece_id=UQHJ4RVNOR7OUXKTAKRUZNBG7LKV2LXRWY7TJMBBKAXTDQ7EMP6A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.480718246 (538h18m24.351801281s) [noise=true , address=145.79.227.45:4230 , piece_id=4V2X6HF6UY2J6LUCWTQ4YMK4R3BEINILJZG2DJ4JNFVJQA4SN7TQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.482376363 (538h18m24.353459398s) [noise=true , address=145.79.227.41:4087 , piece_id=LCQHAZEAM6YZL7UEV6BH2R6SAIL4KB2QPHZZ7RBSXYJA6RYV3IBQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.495507397 (538h18m24.366590442s) [noise=true , address=145.79.227.51:4409 , piece_id=5TADRJVLZWLTRXVEJCE247KFCVF6FDQBU6R4HEKJC35KPJPDSXBA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.511878420 (538h18m24.382961455s) [noise=true , address=145.79.227.47:4267 , piece_id=VWDHNYPGB5ETN2Y444VEXM5R2BRCK63ASUZFHIP3SDTS2DBKVBCA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.745649270 (538h18m24.616732315s) [noise=true , address=77.90.31.12:4590 , piece_id=DMTHXF4ABIIJHDHLAYQBH3QIURA4HVVHV3G4AG6KL4KGCVOLVMKQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.754944830 (538h18m24.626027875s) [noise=true , address=45.91.116.5:4297 , piece_id=QAHSEFPCY3WFAZC7PVNH4XPQO46522XNNHP7RIWH4HBEMXP2CAYQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.763449264 (538h18m24.634532299s) [noise=true , address=77.90.31.10:4518 , piece_id=IHGO3XDH2C6PCCWMGW2FFNSWSL45P4WFO4V755CLW2RI243FGCEQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.771660770 (538h18m24.642743825s) [noise=true , address=45.91.116.4:4937 , piece_id=VLLHKMFCHB5UNNMOJVZCOJVVL2Q2H7DXB3KX3DMDBJK7KBO7OQDQ, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.788420704 (538h18m24.659503739s) [noise=true , address=45.91.116.14:4191 , piece_id=JUOJBAQPTYDDVTHJZDBLQQSDOYIGFOGYVBWMYCOWCU3IWD2FV32A, seg_pos={0 0}, upload=472098]: Done uploading piece. err: <nil> Dec 28 01:47:24.788452504 (538h18m24.659535539s) [seg_pos={0 0}, upload=472098]: Segment reached optimal threshold of 13 pieces. Dec 28 01:47:24.788651243 (538h18m24.659734278s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->77.90.31.14:4662: use of closed network connection Dec 28 01:47:24.788657454 (538h18m24.659740489s) [noise=true , address=77.90.31.14:4662 , piece_id=B4XLS2HSXBQI7QOJM7VSOOFC5TARK2HAIWJUYEVTTD7OXLADQUCA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788840303 (538h18m24.659923348s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49166->145.79.227.3:4151: use of closed network connection Dec 28 01:47:24.788860261 (538h18m24.659943296s) [noise=true , address=145.79.227.3:4151 , piece_id=WASEUMS6EQEJ5TP7GMFELVL24QLNPVUAO2QNA3Q3SGPT5DJTQ5TA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788865641 (538h18m24.659948676s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:47:24.788872645 (538h18m24.65995568s) [noise=true , address=45.91.116.6:4973 , piece_id=ZZ35EXNZGNJQKRBHVLNNQUD4QJELX4IBUVHLN7QM2LJYCAAEIKOA, seg_pos={0 0}, upload=472098]: Long tail cancellation due to optimal threshold Dec 28 01:47:24.788898263 (538h18m24.659981298s) [seg_pos={0 0}, upload=472098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:47:24.788991481 (538h18m24.660074516s) [upload=472098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:47:24.788999416 (538h18m24.660082451s) [upload=472098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:47:25.074744052 (538h18m24.945827097s) [upload=472098]: Done uploading segments... Dec 28 01:47:25.074762378 (538h18m24.945845413s) [upload=472098]: Upload finished. err: <nil> Duration: 30.002527494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.X] duration: 30.003540059s trace: https://jaeger.dev.storj.io/trace/1ec502c9d5159c9a FAIL (2025-12-28 01:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WP7FBS5AUWKVE4NKTS6WCD3TMTKQVSU6IXJMQYFITNFHHBQNKQWQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 4HCYLQO25IZ7GGIGVV2VC2DISRLXUKH73MC5AVU4CEP3KDXZYTNA) context deadline exceeded ======LOGS====== Dec 28 01:46:25.742905759 (538h17m25.613988804s) [upload=472082]: Starting upload Dec 28 01:46:25.742987975 (538h17m25.61407102s) [upload=472082]: Done starting upload Dec 28 01:46:25.743116581 (538h17m25.614199626s) [upload=472082]: Uploading segments... Dec 28 01:46:25.743258881 (538h17m25.614341926s) [upload=472082]: Got next segment. Inline: false Dec 28 01:46:25.743316732 (538h17m25.614399767s) [upload=472082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:46:25.743322743 (538h17m25.614405778s) [upload=472082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:46:25.986565789 (538h17m25.857648824s) [seg_pos={0 0}, upload=472082]: Begin upload segment... Dec 28 01:46:25.986608672 (538h17m25.857691707s) [seg_pos={0 0}, upload=472082]: Done begin upload segment. Dec 28 01:46:25.986635121 (538h17m25.857718156s) [noise=true , address=45.91.116.4:4940 , piece_id=ZPXGWAQBKK536UUMRUGSF5NDTLQYTMQ4F6ZHF5OQYYKW77SMA6PA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.986983224 (538h17m25.858066269s) [noise=true , address=45.91.116.7:4009 , piece_id=4HCYLQO25IZ7GGIGVV2VC2DISRLXUKH73MC5AVU4CEP3KDXZYTNA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.987542179 (538h17m25.858625225s) [noise=true , address=45.91.116.10:4084 , piece_id=3G4NRUV4YYQIKMNTS2X77YVIWZB5FYV5OOFG3C5B2RX7TIDE2D4A, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.988712377 (538h17m25.859795412s) [noise=true , address=77.90.31.13:4627 , piece_id=WP7FBS5AUWKVE4NKTS6WCD3TMTKQVSU6IXJMQYFITNFHHBQNKQWQ, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.988839800 (538h17m25.859922835s) [noise=true , address=45.91.116.11:4120 , piece_id=NUKUUII3R3MDVBCPX7XDCTUZCUNG5MZRLVDQ6745KZLATXAVDM6Q, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.989183034 (538h17m25.860266079s) [noise=true , address=145.79.227.12:4765 , piece_id=VZCXSYISVTY2VVFLL4L5PFUCSYX5AK55LPSTTF3WBJ6EV2AWBPXQ, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.989582183 (538h17m25.860665218s) [noise=true , address=145.79.227.46:4601 , piece_id=W6XSXR26N3XQ3S34MMM56DW4TR7GTJUWPZVJ263W3YDR7XXXTX6A, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.989857317 (538h17m25.860940352s) [noise=true , address=45.91.116.6:4973 , piece_id=VIFXU26XMJ7QGRLRNJYBITU64BW7APHK4UXG2MOCAAWDGSR5PLWA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.990138914 (538h17m25.861221949s) [noise=true , address=145.79.227.50:4373 , piece_id=CSRAP7NDLJXWDUJYIZAUVNSLQHBVCUYK7U2YMSEO7INSML6FSPGQ, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.990808548 (538h17m25.861891593s) [noise=true , address=45.91.116.9:4263 , piece_id=4EMCNVJBYRMDJ2CQQJH4JDMV5UCNMTOU6T4SH72BYZZOHVJFA4BA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.990987079 (538h17m25.862070114s) [noise=true , address=145.79.227.49:4338 , piece_id=Y2QJS4DJYLYQWEXIPGWXXIHZBNJRI6EYBWIFJ3YB5Z5PASR4U5IQ, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.991439019 (538h17m25.862522064s) [noise=true , address=145.79.227.39:4015 , piece_id=CUSS42PSLPQFN65XW4W4YSRHQRGF5E4CVPW3HPRKYKTLCOURTBGQ, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.991568786 (538h17m25.862651821s) [noise=true , address=145.79.227.44:4195 , piece_id=DANHJS6DRSTCK25BZXL7FZLRBP3JY62TE66LV2AT5QKUATCWEOUA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.992114065 (538h17m25.86319711s) [noise=true , address=145.79.227.41:4085 , piece_id=FUAGT6SHB5WYGLD6YWZEVIFFN3HUORCZLMEF5ZQJVID6NB6JFD2Q, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.992163148 (538h17m25.863246183s) [noise=true , address=145.79.227.3:4151 , piece_id=73V7BL5AR7Y7A545K276SJ56RX6YFOH3S3N3XVZNIB7FGONKZBCA, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:25.992716101 (538h17m25.863799136s) [noise=true , address=145.79.227.47:4265 , piece_id=I3L34N2HRIHE66ZRRJJFJW23EEUBLIIZO4TQTSYXRH5XUY4COI2A, seg_pos={0 0}, upload=472082]: Uploading piece... Dec 28 01:46:27.604870289 (538h17m27.475953324s) [upload=472082]: Next returned nil segment Dec 28 01:46:27.604883944 (538h17m27.475966979s) [upload=472082]: Waiting for error group managing segments... Dec 28 01:46:27.989514297 (538h17m27.860597332s) [noise=true , address=145.79.227.12:4765 , piece_id=VZCXSYISVTY2VVFLL4L5PFUCSYX5AK55LPSTTF3WBJ6EV2AWBPXQ, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:27.993488965 (538h17m27.86457201s) [noise=true , address=145.79.227.50:4373 , piece_id=CSRAP7NDLJXWDUJYIZAUVNSLQHBVCUYK7U2YMSEO7INSML6FSPGQ, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:27.995303030 (538h17m27.866386065s) [noise=true , address=145.79.227.41:4085 , piece_id=FUAGT6SHB5WYGLD6YWZEVIFFN3HUORCZLMEF5ZQJVID6NB6JFD2Q, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:27.996087884 (538h17m27.867170929s) [noise=true , address=145.79.227.46:4601 , piece_id=W6XSXR26N3XQ3S34MMM56DW4TR7GTJUWPZVJ263W3YDR7XXXTX6A, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:27.998207891 (538h17m27.869290926s) [noise=true , address=145.79.227.3:4151 , piece_id=73V7BL5AR7Y7A545K276SJ56RX6YFOH3S3N3XVZNIB7FGONKZBCA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.022871225 (538h17m27.89395427s) [noise=true , address=145.79.227.39:4015 , piece_id=CUSS42PSLPQFN65XW4W4YSRHQRGF5E4CVPW3HPRKYKTLCOURTBGQ, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.031254257 (538h17m27.902337292s) [noise=true , address=145.79.227.47:4265 , piece_id=I3L34N2HRIHE66ZRRJJFJW23EEUBLIIZO4TQTSYXRH5XUY4COI2A, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.280016583 (538h17m28.151099628s) [noise=true , address=45.91.116.7:4009 , piece_id=4HCYLQO25IZ7GGIGVV2VC2DISRLXUKH73MC5AVU4CEP3KDXZYTNA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.286817383 (538h17m28.157900418s) [noise=true , address=77.90.31.13:4627 , piece_id=WP7FBS5AUWKVE4NKTS6WCD3TMTKQVSU6IXJMQYFITNFHHBQNKQWQ, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.304305333 (538h17m28.175388368s) [noise=true , address=45.91.116.9:4263 , piece_id=4EMCNVJBYRMDJ2CQQJH4JDMV5UCNMTOU6T4SH72BYZZOHVJFA4BA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:28.403610306 (538h17m28.274693342s) [noise=true , address=45.91.116.11:4120 , piece_id=NUKUUII3R3MDVBCPX7XDCTUZCUNG5MZRLVDQ6745KZLATXAVDM6Q, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:29.066929180 (538h17m28.938012215s) [noise=true , address=45.91.116.10:4084 , piece_id=3G4NRUV4YYQIKMNTS2X77YVIWZB5FYV5OOFG3C5B2RX7TIDE2D4A, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:29.209542540 (538h17m29.080625595s) [noise=true , address=145.79.227.44:4195 , piece_id=DANHJS6DRSTCK25BZXL7FZLRBP3JY62TE66LV2AT5QKUATCWEOUA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: <nil> Dec 28 01:46:29.209571415 (538h17m29.08065445s) [seg_pos={0 0}, upload=472082]: Segment reached optimal threshold of 13 pieces. Dec 28 01:46:29.209844635 (538h17m29.08092768s) [noise=true , address=145.79.227.49:4338 , piece_id=Y2QJS4DJYLYQWEXIPGWXXIHZBNJRI6EYBWIFJ3YB5Z5PASR4U5IQ, seg_pos={0 0}, upload=472082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 01:46:29.209854173 (538h17m29.080937218s) [noise=true , address=145.79.227.49:4338 , piece_id=Y2QJS4DJYLYQWEXIPGWXXIHZBNJRI6EYBWIFJ3YB5Z5PASR4U5IQ, seg_pos={0 0}, upload=472082]: Long tail cancellation due to optimal threshold Dec 28 01:46:29.209876606 (538h17m29.080959651s) [noise=true , address=45.91.116.6:4973 , piece_id=VIFXU26XMJ7QGRLRNJYBITU64BW7APHK4UXG2MOCAAWDGSR5PLWA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 01:46:29.209888228 (538h17m29.080971263s) [noise=true , address=45.91.116.6:4973 , piece_id=VIFXU26XMJ7QGRLRNJYBITU64BW7APHK4UXG2MOCAAWDGSR5PLWA, seg_pos={0 0}, upload=472082]: Long tail cancellation due to optimal threshold Dec 28 01:46:29.210200242 (538h17m29.081283277s) [noise=true , address=45.91.116.4:4940 , piece_id=ZPXGWAQBKK536UUMRUGSF5NDTLQYTMQ4F6ZHF5OQYYKW77SMA6PA, seg_pos={0 0}, upload=472082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:37554->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:37554->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37554->45.91.116.4:4940: use of closed network connection Dec 28 01:46:29.210210602 (538h17m29.081293647s) [noise=true , address=45.91.116.4:4940 , piece_id=ZPXGWAQBKK536UUMRUGSF5NDTLQYTMQ4F6ZHF5OQYYKW77SMA6PA, seg_pos={0 0}, upload=472082]: Long tail cancellation due to optimal threshold Dec 28 01:46:29.210233766 (538h17m29.081316811s) [seg_pos={0 0}, upload=472082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:46:29.210383230 (538h17m29.081466276s) [upload=472082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:46:29.210390855 (538h17m29.0814739s) [upload=472082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:46:29.500004148 (538h17m29.371087193s) [upload=472082]: Done uploading segments... Dec 28 01:46:29.500023736 (538h17m29.371106771s) [upload=472082]: Upload finished. err: <nil> Duration: 30.003560508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X.] duration: 20.779006616s trace: https://jaeger.dev.storj.io/trace/77fcbd5ff3e7535d FAIL (2025-12-28 01:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:44:12.222582119 (538h15m12.093665154s) [upload=472048]: Starting upload Dec 28 01:44:12.222634007 (538h15m12.093717052s) [upload=472048]: Done starting upload Dec 28 01:44:12.222883943 (538h15m12.093966978s) [upload=472048]: Uploading segments... Dec 28 01:44:12.223337015 (538h15m12.09442005s) [upload=472048]: Got next segment. Inline: false Dec 28 01:44:12.223439070 (538h15m12.094522105s) [upload=472048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:44:12.223444961 (538h15m12.094527996s) [upload=472048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:44:12.472026273 (538h15m12.343109318s) [seg_pos={0 0}, upload=472048]: Begin upload segment... Dec 28 01:44:12.472084974 (538h15m12.343168009s) [seg_pos={0 0}, upload=472048]: Done begin upload segment. Dec 28 01:44:12.472094513 (538h15m12.343177548s) [noise=true , address=45.91.116.7:4009 , piece_id=6NZQ47JKZCS5DBSSREQ4VGYP4XDVQGWYA3PGV3QG3M72OFGSRBYA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.472125151 (538h15m12.343208196s) [noise=true , address=45.91.116.10:4083 , piece_id=KRAPNOAZAVJQD7R42BQG3VHA5E3S3VAJ2UOZ2QXLIFHI5YZGZKEQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.472914163 (538h15m12.343997198s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473181142 (538h15m12.344264187s) [noise=true , address=145.79.227.45:4230 , piece_id=WMPYNJXQS5GB6QFB2H4SSXOX23JPQJZJ74DCOQL7VQKOLWHO4TFQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473390961 (538h15m12.344473996s) [noise=true , address=145.79.227.15:4875 , piece_id=PU5CFCGKQTC5KN4KLWYQM7363SCRQ5KKFJYUV66IQFJ6PSII55VA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473711531 (538h15m12.344794566s) [noise=true , address=77.90.31.18:4808 , piece_id=IZ474KWZB7D7XLELROMU77M7R3YACWF3L2C7QW42H34R2U2LVISA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.474350898 (538h15m12.345433933s) [noise=true , address=145.79.227.7:4295 , piece_id=IMRARCHJJUM57B2UV77CCGMTEZVRNNHWLTOJ5S2UFBYTBY2UASVA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475131836 (538h15m12.346214881s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475370811 (538h15m12.346453856s) [noise=true , address=145.79.227.49:4339 , piece_id=66QE5DGVHVCLEZ6GZMC3Y6JIIX6NHCNZU2FUNQCLPWCDRLWWJJWA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475767456 (538h15m12.346850491s) [noise=true , address=45.91.116.11:4120 , piece_id=7GCENQOXLVK4TFKLEBUCOU2F33KU5IUKTB4IGTAMLNZCNL5HP77A, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476207635 (538h15m12.34729067s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476484131 (538h15m12.347567166s) [noise=true , address=45.91.116.4:4940 , piece_id=T25ZWZTCKWWS6IDDAISRATLE67KBA3SBIM24NI2YNDDR7PST4UPA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476928988 (538h15m12.348012023s) [noise=true , address=145.79.227.47:4265 , piece_id=QPKSQFIWIGTODEXYKCNYAU6IDM6KNDME775RRVEQ26PCVN7ABWXQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.477374276 (538h15m12.348457311s) [noise=true , address=145.79.227.18:4006 , piece_id=N26OYZKNZCFU6HM5UXXJJIKQ62MFQF2CDWOX6VUA7MDJL3YMWSCQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.478037759 (538h15m12.349120794s) [noise=true , address=145.79.227.46:4603 , piece_id=5H6GS7RKFBXG6P6ILSNENRILCJZRHI2YCHWSZ37BF3DCKDTEQMDA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.479424740 (538h15m12.350507785s) [noise=true , address=145.79.227.14:4840 , piece_id=XBEVEOSBJ4HSMYQV3IW3FK35KLTG7BBRRNSZ3FZOZXPAIMO2SWYA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:14.088320023 (538h15m13.959403068s) [upload=472048]: Next returned nil segment Dec 28 01:44:14.088335293 (538h15m13.959418338s) [upload=472048]: Waiting for error group managing segments... Dec 28 01:44:14.472648560 (538h15m14.343731595s) [noise=true , address=145.79.227.46:4603 , piece_id=5H6GS7RKFBXG6P6ILSNENRILCJZRHI2YCHWSZ37BF3DCKDTEQMDA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.475417964 (538h15m14.346501009s) [noise=true , address=145.79.227.15:4875 , piece_id=PU5CFCGKQTC5KN4KLWYQM7363SCRQ5KKFJYUV66IQFJ6PSII55VA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.477684220 (538h15m14.348767255s) [noise=true , address=145.79.227.14:4840 , piece_id=XBEVEOSBJ4HSMYQV3IW3FK35KLTG7BBRRNSZ3FZOZXPAIMO2SWYA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.481050609 (538h15m14.352133644s) [noise=true , address=145.79.227.49:4339 , piece_id=66QE5DGVHVCLEZ6GZMC3Y6JIIX6NHCNZU2FUNQCLPWCDRLWWJJWA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.483171107 (538h15m14.354254152s) [noise=true , address=145.79.227.7:4295 , piece_id=IMRARCHJJUM57B2UV77CCGMTEZVRNNHWLTOJ5S2UFBYTBY2UASVA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.485713959 (538h15m14.356797004s) [noise=true , address=145.79.227.47:4265 , piece_id=QPKSQFIWIGTODEXYKCNYAU6IDM6KNDME775RRVEQ26PCVN7ABWXQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.496608785 (538h15m14.36769183s) [noise=true , address=145.79.227.18:4006 , piece_id=N26OYZKNZCFU6HM5UXXJJIKQ62MFQF2CDWOX6VUA7MDJL3YMWSCQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.500137845 (538h15m14.37122088s) [noise=true , address=145.79.227.45:4230 , piece_id=WMPYNJXQS5GB6QFB2H4SSXOX23JPQJZJ74DCOQL7VQKOLWHO4TFQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.688265673 (538h15m14.559348718s) [noise=true , address=45.91.116.4:4940 , piece_id=T25ZWZTCKWWS6IDDAISRATLE67KBA3SBIM24NI2YNDDR7PST4UPA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.707489658 (538h15m14.578572693s) [noise=true , address=45.91.116.7:4009 , piece_id=6NZQ47JKZCS5DBSSREQ4VGYP4XDVQGWYA3PGV3QG3M72OFGSRBYA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.755543652 (538h15m14.626626687s) [noise=true , address=45.91.116.11:4120 , piece_id=7GCENQOXLVK4TFKLEBUCOU2F33KU5IUKTB4IGTAMLNZCNL5HP77A, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.836461884 (538h15m14.707544919s) [noise=true , address=45.91.116.10:4083 , piece_id=KRAPNOAZAVJQD7R42BQG3VHA5E3S3VAJ2UOZ2QXLIFHI5YZGZKEQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.844487425 (538h15m14.71557046s) [noise=true , address=77.90.31.18:4808 , piece_id=IZ474KWZB7D7XLELROMU77M7R3YACWF3L2C7QW42H34R2U2LVISA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.844514737 (538h15m14.715597782s) [seg_pos={0 0}, upload=472048]: Segment reached optimal threshold of 13 pieces. Dec 28 01:44:14.844757409 (538h15m14.715840445s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): internal: context canceled Dec 28 01:44:14.844775685 (538h15m14.71585873s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.844790523 (538h15m14.715873568s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 01:44:14.844800050 (538h15m14.715883085s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.844988049 (538h15m14.716071094s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection Dec 28 01:44:14.844996445 (538h15m14.71607949s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.845011664 (538h15m14.716094709s) [seg_pos={0 0}, upload=472048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:44:14.845082229 (538h15m14.716165264s) [upload=472048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:44:14.845086316 (538h15m14.716169361s) [upload=472048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:44:15.150051532 (538h15m15.021134587s) [upload=472048]: Done uploading segments... Dec 28 01:44:15.150078584 (538h15m15.021161629s) [upload=472048]: Upload finished. err: <nil> Duration: 20.779028118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.X] duration: 30.003116104s trace: https://jaeger.dev.storj.io/trace/77fcbd5ff3e7535d FAIL (2025-12-28 01:44:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:44:12.222582119 (538h15m12.093665154s) [upload=472048]: Starting upload Dec 28 01:44:12.222634007 (538h15m12.093717052s) [upload=472048]: Done starting upload Dec 28 01:44:12.222883943 (538h15m12.093966978s) [upload=472048]: Uploading segments... Dec 28 01:44:12.223337015 (538h15m12.09442005s) [upload=472048]: Got next segment. Inline: false Dec 28 01:44:12.223439070 (538h15m12.094522105s) [upload=472048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:44:12.223444961 (538h15m12.094527996s) [upload=472048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:44:12.472026273 (538h15m12.343109318s) [seg_pos={0 0}, upload=472048]: Begin upload segment... Dec 28 01:44:12.472084974 (538h15m12.343168009s) [seg_pos={0 0}, upload=472048]: Done begin upload segment. Dec 28 01:44:12.472094513 (538h15m12.343177548s) [noise=true , address=45.91.116.7:4009 , piece_id=6NZQ47JKZCS5DBSSREQ4VGYP4XDVQGWYA3PGV3QG3M72OFGSRBYA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.472125151 (538h15m12.343208196s) [noise=true , address=45.91.116.10:4083 , piece_id=KRAPNOAZAVJQD7R42BQG3VHA5E3S3VAJ2UOZ2QXLIFHI5YZGZKEQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.472914163 (538h15m12.343997198s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473181142 (538h15m12.344264187s) [noise=true , address=145.79.227.45:4230 , piece_id=WMPYNJXQS5GB6QFB2H4SSXOX23JPQJZJ74DCOQL7VQKOLWHO4TFQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473390961 (538h15m12.344473996s) [noise=true , address=145.79.227.15:4875 , piece_id=PU5CFCGKQTC5KN4KLWYQM7363SCRQ5KKFJYUV66IQFJ6PSII55VA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.473711531 (538h15m12.344794566s) [noise=true , address=77.90.31.18:4808 , piece_id=IZ474KWZB7D7XLELROMU77M7R3YACWF3L2C7QW42H34R2U2LVISA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.474350898 (538h15m12.345433933s) [noise=true , address=145.79.227.7:4295 , piece_id=IMRARCHJJUM57B2UV77CCGMTEZVRNNHWLTOJ5S2UFBYTBY2UASVA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475131836 (538h15m12.346214881s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475370811 (538h15m12.346453856s) [noise=true , address=145.79.227.49:4339 , piece_id=66QE5DGVHVCLEZ6GZMC3Y6JIIX6NHCNZU2FUNQCLPWCDRLWWJJWA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.475767456 (538h15m12.346850491s) [noise=true , address=45.91.116.11:4120 , piece_id=7GCENQOXLVK4TFKLEBUCOU2F33KU5IUKTB4IGTAMLNZCNL5HP77A, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476207635 (538h15m12.34729067s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476484131 (538h15m12.347567166s) [noise=true , address=45.91.116.4:4940 , piece_id=T25ZWZTCKWWS6IDDAISRATLE67KBA3SBIM24NI2YNDDR7PST4UPA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.476928988 (538h15m12.348012023s) [noise=true , address=145.79.227.47:4265 , piece_id=QPKSQFIWIGTODEXYKCNYAU6IDM6KNDME775RRVEQ26PCVN7ABWXQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.477374276 (538h15m12.348457311s) [noise=true , address=145.79.227.18:4006 , piece_id=N26OYZKNZCFU6HM5UXXJJIKQ62MFQF2CDWOX6VUA7MDJL3YMWSCQ, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.478037759 (538h15m12.349120794s) [noise=true , address=145.79.227.46:4603 , piece_id=5H6GS7RKFBXG6P6ILSNENRILCJZRHI2YCHWSZ37BF3DCKDTEQMDA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:12.479424740 (538h15m12.350507785s) [noise=true , address=145.79.227.14:4840 , piece_id=XBEVEOSBJ4HSMYQV3IW3FK35KLTG7BBRRNSZ3FZOZXPAIMO2SWYA, seg_pos={0 0}, upload=472048]: Uploading piece... Dec 28 01:44:14.088320023 (538h15m13.959403068s) [upload=472048]: Next returned nil segment Dec 28 01:44:14.088335293 (538h15m13.959418338s) [upload=472048]: Waiting for error group managing segments... Dec 28 01:44:14.472648560 (538h15m14.343731595s) [noise=true , address=145.79.227.46:4603 , piece_id=5H6GS7RKFBXG6P6ILSNENRILCJZRHI2YCHWSZ37BF3DCKDTEQMDA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.475417964 (538h15m14.346501009s) [noise=true , address=145.79.227.15:4875 , piece_id=PU5CFCGKQTC5KN4KLWYQM7363SCRQ5KKFJYUV66IQFJ6PSII55VA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.477684220 (538h15m14.348767255s) [noise=true , address=145.79.227.14:4840 , piece_id=XBEVEOSBJ4HSMYQV3IW3FK35KLTG7BBRRNSZ3FZOZXPAIMO2SWYA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.481050609 (538h15m14.352133644s) [noise=true , address=145.79.227.49:4339 , piece_id=66QE5DGVHVCLEZ6GZMC3Y6JIIX6NHCNZU2FUNQCLPWCDRLWWJJWA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.483171107 (538h15m14.354254152s) [noise=true , address=145.79.227.7:4295 , piece_id=IMRARCHJJUM57B2UV77CCGMTEZVRNNHWLTOJ5S2UFBYTBY2UASVA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.485713959 (538h15m14.356797004s) [noise=true , address=145.79.227.47:4265 , piece_id=QPKSQFIWIGTODEXYKCNYAU6IDM6KNDME775RRVEQ26PCVN7ABWXQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.496608785 (538h15m14.36769183s) [noise=true , address=145.79.227.18:4006 , piece_id=N26OYZKNZCFU6HM5UXXJJIKQ62MFQF2CDWOX6VUA7MDJL3YMWSCQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.500137845 (538h15m14.37122088s) [noise=true , address=145.79.227.45:4230 , piece_id=WMPYNJXQS5GB6QFB2H4SSXOX23JPQJZJ74DCOQL7VQKOLWHO4TFQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.688265673 (538h15m14.559348718s) [noise=true , address=45.91.116.4:4940 , piece_id=T25ZWZTCKWWS6IDDAISRATLE67KBA3SBIM24NI2YNDDR7PST4UPA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.707489658 (538h15m14.578572693s) [noise=true , address=45.91.116.7:4009 , piece_id=6NZQ47JKZCS5DBSSREQ4VGYP4XDVQGWYA3PGV3QG3M72OFGSRBYA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.755543652 (538h15m14.626626687s) [noise=true , address=45.91.116.11:4120 , piece_id=7GCENQOXLVK4TFKLEBUCOU2F33KU5IUKTB4IGTAMLNZCNL5HP77A, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.836461884 (538h15m14.707544919s) [noise=true , address=45.91.116.10:4083 , piece_id=KRAPNOAZAVJQD7R42BQG3VHA5E3S3VAJ2UOZ2QXLIFHI5YZGZKEQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.844487425 (538h15m14.71557046s) [noise=true , address=77.90.31.18:4808 , piece_id=IZ474KWZB7D7XLELROMU77M7R3YACWF3L2C7QW42H34R2U2LVISA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: <nil> Dec 28 01:44:14.844514737 (538h15m14.715597782s) [seg_pos={0 0}, upload=472048]: Segment reached optimal threshold of 13 pieces. Dec 28 01:44:14.844757409 (538h15m14.715840445s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): internal: context canceled Dec 28 01:44:14.844775685 (538h15m14.71585873s) [noise=true , address=77.90.31.10:4520 , piece_id=GYEWDEQQ7BZWBRRZG5LAQ6O63HDSBBRIFIQTPC3SO4AFOUAAXDHQ, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.844790523 (538h15m14.715873568s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 01:44:14.844800050 (538h15m14.715883085s) [noise=true , address=45.91.116.5:4300 , piece_id=ZO2YEH5LQIAZXPAMW42FGLTC5HBNUJCAO4N2ZWKXG26H6BMWFEIA, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.844988049 (538h15m14.716071094s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.11:4554: use of closed network connection Dec 28 01:44:14.844996445 (538h15m14.71607949s) [noise=true , address=77.90.31.11:4554 , piece_id=63NFX5AAZEMBINGKVCQ3XTQFTFAWIQCFHJQOXEGZ7JN7HNWVJF5Q, seg_pos={0 0}, upload=472048]: Long tail cancellation due to optimal threshold Dec 28 01:44:14.845011664 (538h15m14.716094709s) [seg_pos={0 0}, upload=472048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:44:14.845082229 (538h15m14.716165264s) [upload=472048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:44:14.845086316 (538h15m14.716169361s) [upload=472048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:44:15.150051532 (538h15m15.021134587s) [upload=472048]: Done uploading segments... Dec 28 01:44:15.150078584 (538h15m15.021161629s) [upload=472048]: Upload finished. err: <nil> Duration: 30.00314078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX.] duration: 6.398462561s trace: https://jaeger.dev.storj.io/trace/75a2c6111b938188 FAIL (2025-12-28 01:43:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:42:19.772963698 (538h13m19.644046743s) [upload=472021]: Starting upload Dec 28 01:42:19.773035585 (538h13m19.64411862s) [upload=472021]: Done starting upload Dec 28 01:42:19.773140194 (538h13m19.644223229s) [upload=472021]: Uploading segments... Dec 28 01:42:19.773199237 (538h13m19.644282272s) [upload=472021]: Got next segment. Inline: false Dec 28 01:42:19.773228473 (538h13m19.644311508s) [upload=472021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:42:19.773233172 (538h13m19.644316217s) [upload=472021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:42:20.133351961 (538h13m20.004434996s) [seg_pos={0 0}, upload=472021]: Begin upload segment... Dec 28 01:42:20.133406995 (538h13m20.00449004s) [seg_pos={0 0}, upload=472021]: Done begin upload segment. Dec 28 01:42:20.133443856 (538h13m20.004526891s) [noise=true , address=145.79.227.5:4222 , piece_id=I6E6BZYOVKY6MCELLGXBIALLMJPKXEPGPJ3BPIE6KEUMALBRJJAQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.133538906 (538h13m20.004621952s) [noise=true , address=145.79.227.36:4518 , piece_id=7432ZNI5G3D4C5VTERB5ZMQQKSJ2Y7PGSGY47I6LR2XBHMTPYFAA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.134191850 (538h13m20.005274895s) [noise=true , address=145.79.227.47:4265 , piece_id=CBNHC54UARGEBCSMWICSSAURAFDR4WEEQJLGEQXG34EELNLOYSIA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.134769661 (538h13m20.005852706s) [noise=true , address=145.79.227.51:4411 , piece_id=LEHSKWJOQJGFYDRZZZN3FBDMAKHQZEF7GQFSAQNMMS6J2TAJH6FA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135202374 (538h13m20.006285419s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135623035 (538h13m20.00670607s) [noise=true , address=77.90.31.13:4629 , piece_id=P374OKU5GYX4IQA4KMOSQK5BPQDAUF4LUQQBJ726BFRTKUPZDBXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135631661 (538h13m20.006714706s) [noise=true , address=45.91.116.6:4973 , piece_id=J6KX3GL5EWA2ID2AZYPTMJMJG3ZYMBEUJ4MRHOMA245SP63LBMQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136086588 (538h13m20.007169623s) [noise=true , address=145.79.227.19:4077 , piece_id=MQULM6KI3MFPXMS6LETTHVSEI37H5BGUSHTETJBW5WL23J7JEQMA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136101126 (538h13m20.007184161s) [noise=true , address=145.79.227.11:4731 , piece_id=U4F662KIVFPEMF3ULHWU27YMC34CGHH5C2FI5WCVQFKBVQDN4JQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136561132 (538h13m20.007644167s) [noise=true , address=45.91.116.7:4012 , piece_id=COSMNPEMVLCKQ443RP3W7WL24OSBKBEBVA47XOOR336LROWLJBQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136582883 (538h13m20.007665928s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.137067807 (538h13m20.008150852s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.137080811 (538h13m20.008163846s) [noise=true , address=145.79.227.3:4151 , piece_id=K5DB56JC2MFT6TOJEUZ5XDOTBTOZUHZ5WJGWLH5PF3UB3HG33ZVQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.139161784 (538h13m20.010244829s) [noise=true , address=45.91.116.5:4299 , piece_id=45FDKRB7ANQXL5OS6EBA3MLFFLUFU2YNUQ34DJYWTKXRXU5RYSFQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.140195692 (538h13m20.011278737s) [noise=true , address=145.79.227.13:4803 , piece_id=4LK3FWTF3GPVHHSG3ZZTM7RPU2ZUAA52DY42OJBEIFEFDWJNL54A, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.140281726 (538h13m20.011364841s) [noise=true , address=145.79.227.31:4798 , piece_id=6CTDAZ6PBOZZK2Y2KE3WDBRZ6TDOHAHS6QJE67A6J5ITON3ITDXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:21.838255762 (538h13m21.709338797s) [upload=472021]: Next returned nil segment Dec 28 01:42:21.838271491 (538h13m21.709354546s) [upload=472021]: Waiting for error group managing segments... Dec 28 01:42:22.145218400 (538h13m22.016301506s) [noise=true , address=145.79.227.5:4222 , piece_id=I6E6BZYOVKY6MCELLGXBIALLMJPKXEPGPJ3BPIE6KEUMALBRJJAQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.154497730 (538h13m22.025580765s) [noise=true , address=145.79.227.11:4731 , piece_id=U4F662KIVFPEMF3ULHWU27YMC34CGHH5C2FI5WCVQFKBVQDN4JQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.167925700 (538h13m22.039008745s) [noise=true , address=145.79.227.13:4803 , piece_id=4LK3FWTF3GPVHHSG3ZZTM7RPU2ZUAA52DY42OJBEIFEFDWJNL54A, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.184166394 (538h13m22.055249429s) [noise=true , address=145.79.227.47:4265 , piece_id=CBNHC54UARGEBCSMWICSSAURAFDR4WEEQJLGEQXG34EELNLOYSIA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.194266827 (538h13m22.065349862s) [noise=true , address=145.79.227.31:4798 , piece_id=6CTDAZ6PBOZZK2Y2KE3WDBRZ6TDOHAHS6QJE67A6J5ITON3ITDXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.213523937 (538h13m22.084606972s) [noise=true , address=145.79.227.3:4151 , piece_id=K5DB56JC2MFT6TOJEUZ5XDOTBTOZUHZ5WJGWLH5PF3UB3HG33ZVQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.418420976 (538h13m22.289504021s) [noise=true , address=77.90.31.13:4629 , piece_id=P374OKU5GYX4IQA4KMOSQK5BPQDAUF4LUQQBJ726BFRTKUPZDBXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.470925202 (538h13m22.342008257s) [noise=true , address=45.91.116.7:4012 , piece_id=COSMNPEMVLCKQ443RP3W7WL24OSBKBEBVA47XOOR336LROWLJBQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.501434938 (538h13m22.372517973s) [noise=true , address=45.91.116.6:4973 , piece_id=J6KX3GL5EWA2ID2AZYPTMJMJG3ZYMBEUJ4MRHOMA245SP63LBMQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.627805880 (538h13m22.498888915s) [noise=true , address=45.91.116.5:4299 , piece_id=45FDKRB7ANQXL5OS6EBA3MLFFLUFU2YNUQ34DJYWTKXRXU5RYSFQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.084114941 (538h13m22.955197986s) [noise=true , address=145.79.227.51:4411 , piece_id=LEHSKWJOQJGFYDRZZZN3FBDMAKHQZEF7GQFSAQNMMS6J2TAJH6FA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.096777383 (538h13m22.967860418s) [noise=true , address=145.79.227.19:4077 , piece_id=MQULM6KI3MFPXMS6LETTHVSEI37H5BGUSHTETJBW5WL23J7JEQMA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.100193357 (538h13m22.971276392s) [noise=true , address=145.79.227.36:4518 , piece_id=7432ZNI5G3D4C5VTERB5ZMQQKSJ2Y7PGSGY47I6LR2XBHMTPYFAA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.100208847 (538h13m22.971291882s) [seg_pos={0 0}, upload=472021]: Segment reached optimal threshold of 13 pieces. Dec 28 01:42:23.100373781 (538h13m22.971456826s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection Dec 28 01:42:23.100379251 (538h13m22.971462286s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100521192 (538h13m22.971604227s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 28 01:42:23.100530680 (538h13m22.971613725s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100548984 (538h13m22.971632019s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 01:42:23.100555186 (538h13m22.971638221s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100638595 (538h13m22.97172164s) [seg_pos={0 0}, upload=472021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:42:23.100715912 (538h13m22.971798947s) [upload=472021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:42:23.100720220 (538h13m22.971803255s) [upload=472021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:42:23.486747503 (538h13m23.357830548s) [upload=472021]: Done uploading segments... Dec 28 01:42:23.486773011 (538h13m23.357856046s) [upload=472021]: Upload finished. err: <nil> Duration: 6.398486677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XXX] duration: 30.002263942s trace: https://jaeger.dev.storj.io/trace/75a2c6111b938188 FAIL (2025-12-28 01:42:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:42:19.772963698 (538h13m19.644046743s) [upload=472021]: Starting upload Dec 28 01:42:19.773035585 (538h13m19.64411862s) [upload=472021]: Done starting upload Dec 28 01:42:19.773140194 (538h13m19.644223229s) [upload=472021]: Uploading segments... Dec 28 01:42:19.773199237 (538h13m19.644282272s) [upload=472021]: Got next segment. Inline: false Dec 28 01:42:19.773228473 (538h13m19.644311508s) [upload=472021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:42:19.773233172 (538h13m19.644316217s) [upload=472021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:42:20.133351961 (538h13m20.004434996s) [seg_pos={0 0}, upload=472021]: Begin upload segment... Dec 28 01:42:20.133406995 (538h13m20.00449004s) [seg_pos={0 0}, upload=472021]: Done begin upload segment. Dec 28 01:42:20.133443856 (538h13m20.004526891s) [noise=true , address=145.79.227.5:4222 , piece_id=I6E6BZYOVKY6MCELLGXBIALLMJPKXEPGPJ3BPIE6KEUMALBRJJAQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.133538906 (538h13m20.004621952s) [noise=true , address=145.79.227.36:4518 , piece_id=7432ZNI5G3D4C5VTERB5ZMQQKSJ2Y7PGSGY47I6LR2XBHMTPYFAA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.134191850 (538h13m20.005274895s) [noise=true , address=145.79.227.47:4265 , piece_id=CBNHC54UARGEBCSMWICSSAURAFDR4WEEQJLGEQXG34EELNLOYSIA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.134769661 (538h13m20.005852706s) [noise=true , address=145.79.227.51:4411 , piece_id=LEHSKWJOQJGFYDRZZZN3FBDMAKHQZEF7GQFSAQNMMS6J2TAJH6FA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135202374 (538h13m20.006285419s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135623035 (538h13m20.00670607s) [noise=true , address=77.90.31.13:4629 , piece_id=P374OKU5GYX4IQA4KMOSQK5BPQDAUF4LUQQBJ726BFRTKUPZDBXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.135631661 (538h13m20.006714706s) [noise=true , address=45.91.116.6:4973 , piece_id=J6KX3GL5EWA2ID2AZYPTMJMJG3ZYMBEUJ4MRHOMA245SP63LBMQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136086588 (538h13m20.007169623s) [noise=true , address=145.79.227.19:4077 , piece_id=MQULM6KI3MFPXMS6LETTHVSEI37H5BGUSHTETJBW5WL23J7JEQMA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136101126 (538h13m20.007184161s) [noise=true , address=145.79.227.11:4731 , piece_id=U4F662KIVFPEMF3ULHWU27YMC34CGHH5C2FI5WCVQFKBVQDN4JQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136561132 (538h13m20.007644167s) [noise=true , address=45.91.116.7:4012 , piece_id=COSMNPEMVLCKQ443RP3W7WL24OSBKBEBVA47XOOR336LROWLJBQQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.136582883 (538h13m20.007665928s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.137067807 (538h13m20.008150852s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.137080811 (538h13m20.008163846s) [noise=true , address=145.79.227.3:4151 , piece_id=K5DB56JC2MFT6TOJEUZ5XDOTBTOZUHZ5WJGWLH5PF3UB3HG33ZVQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.139161784 (538h13m20.010244829s) [noise=true , address=45.91.116.5:4299 , piece_id=45FDKRB7ANQXL5OS6EBA3MLFFLUFU2YNUQ34DJYWTKXRXU5RYSFQ, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.140195692 (538h13m20.011278737s) [noise=true , address=145.79.227.13:4803 , piece_id=4LK3FWTF3GPVHHSG3ZZTM7RPU2ZUAA52DY42OJBEIFEFDWJNL54A, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:20.140281726 (538h13m20.011364841s) [noise=true , address=145.79.227.31:4798 , piece_id=6CTDAZ6PBOZZK2Y2KE3WDBRZ6TDOHAHS6QJE67A6J5ITON3ITDXA, seg_pos={0 0}, upload=472021]: Uploading piece... Dec 28 01:42:21.838255762 (538h13m21.709338797s) [upload=472021]: Next returned nil segment Dec 28 01:42:21.838271491 (538h13m21.709354546s) [upload=472021]: Waiting for error group managing segments... Dec 28 01:42:22.145218400 (538h13m22.016301506s) [noise=true , address=145.79.227.5:4222 , piece_id=I6E6BZYOVKY6MCELLGXBIALLMJPKXEPGPJ3BPIE6KEUMALBRJJAQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.154497730 (538h13m22.025580765s) [noise=true , address=145.79.227.11:4731 , piece_id=U4F662KIVFPEMF3ULHWU27YMC34CGHH5C2FI5WCVQFKBVQDN4JQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.167925700 (538h13m22.039008745s) [noise=true , address=145.79.227.13:4803 , piece_id=4LK3FWTF3GPVHHSG3ZZTM7RPU2ZUAA52DY42OJBEIFEFDWJNL54A, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.184166394 (538h13m22.055249429s) [noise=true , address=145.79.227.47:4265 , piece_id=CBNHC54UARGEBCSMWICSSAURAFDR4WEEQJLGEQXG34EELNLOYSIA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.194266827 (538h13m22.065349862s) [noise=true , address=145.79.227.31:4798 , piece_id=6CTDAZ6PBOZZK2Y2KE3WDBRZ6TDOHAHS6QJE67A6J5ITON3ITDXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.213523937 (538h13m22.084606972s) [noise=true , address=145.79.227.3:4151 , piece_id=K5DB56JC2MFT6TOJEUZ5XDOTBTOZUHZ5WJGWLH5PF3UB3HG33ZVQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.418420976 (538h13m22.289504021s) [noise=true , address=77.90.31.13:4629 , piece_id=P374OKU5GYX4IQA4KMOSQK5BPQDAUF4LUQQBJ726BFRTKUPZDBXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.470925202 (538h13m22.342008257s) [noise=true , address=45.91.116.7:4012 , piece_id=COSMNPEMVLCKQ443RP3W7WL24OSBKBEBVA47XOOR336LROWLJBQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.501434938 (538h13m22.372517973s) [noise=true , address=45.91.116.6:4973 , piece_id=J6KX3GL5EWA2ID2AZYPTMJMJG3ZYMBEUJ4MRHOMA245SP63LBMQQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:22.627805880 (538h13m22.498888915s) [noise=true , address=45.91.116.5:4299 , piece_id=45FDKRB7ANQXL5OS6EBA3MLFFLUFU2YNUQ34DJYWTKXRXU5RYSFQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.084114941 (538h13m22.955197986s) [noise=true , address=145.79.227.51:4411 , piece_id=LEHSKWJOQJGFYDRZZZN3FBDMAKHQZEF7GQFSAQNMMS6J2TAJH6FA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.096777383 (538h13m22.967860418s) [noise=true , address=145.79.227.19:4077 , piece_id=MQULM6KI3MFPXMS6LETTHVSEI37H5BGUSHTETJBW5WL23J7JEQMA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.100193357 (538h13m22.971276392s) [noise=true , address=145.79.227.36:4518 , piece_id=7432ZNI5G3D4C5VTERB5ZMQQKSJ2Y7PGSGY47I6LR2XBHMTPYFAA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: <nil> Dec 28 01:42:23.100208847 (538h13m22.971291882s) [seg_pos={0 0}, upload=472021]: Segment reached optimal threshold of 13 pieces. Dec 28 01:42:23.100373781 (538h13m22.971456826s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54638->145.79.227.16:4912: use of closed network connection Dec 28 01:42:23.100379251 (538h13m22.971462286s) [noise=true , address=145.79.227.16:4912 , piece_id=J6INBLO7YBQLZOBPMCOPC7XWQUTOEFSU7NSDRQJQI7GCGAX666BQ, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100521192 (538h13m22.971604227s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 28 01:42:23.100530680 (538h13m22.971613725s) [noise=true , address=145.79.227.29:4942 , piece_id=XXSOHUWP4IY2V66B6BXPU4E7JCZ6NIHEJZN34UWNBTZMJ3GFXHZA, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100548984 (538h13m22.971632019s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 28 01:42:23.100555186 (538h13m22.971638221s) [noise=true , address=145.79.227.4:4185 , piece_id=2L6WFGH3A4RT7GZGFBSZM3H3TYE2LYBQEBPUHDPFUZ7IX7S44IXA, seg_pos={0 0}, upload=472021]: Long tail cancellation due to optimal threshold Dec 28 01:42:23.100638595 (538h13m22.97172164s) [seg_pos={0 0}, upload=472021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:42:23.100715912 (538h13m22.971798947s) [upload=472021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:42:23.100720220 (538h13m22.971803255s) [upload=472021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:42:23.486747503 (538h13m23.357830548s) [upload=472021]: Done uploading segments... Dec 28 01:42:23.486773011 (538h13m23.357856046s) [upload=472021]: Upload finished. err: <nil> Duration: 30.00228931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.XX] duration: 30.003464733s trace: https://jaeger.dev.storj.io/trace/63e107dc36546fdf FAIL (2025-12-28 01:41:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VFEA4TQKMSRWYA3ONC55MJMIQFYPDS4KWS6FVKC4BYP5ZXHUIGHQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: UNGVIGJ7KIOVQTDZQEX5NOMT7NWIVQ44M4OBEGARSHQOQQMNFFAA) context deadline exceeded ======LOGS====== h12m24.716270185s) [upload=472008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:41:24.845193672 (538h12m24.716276717s) [upload=472008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:41:25.102546576 (538h12m24.973629621s) [seg_pos={0 0}, upload=472008]: Begin upload segment... Dec 28 01:41:25.102593034 (538h12m24.973676069s) [seg_pos={0 0}, upload=472008]: Done begin upload segment. Dec 28 01:41:25.102619986 (538h12m24.973703021s) [noise=true , address=145.79.227.19:4077 , piece_id=GBL3ZP7LUZCS64MOOUOESRXE3TGCEMCSLJY266TZKZ4PPPRG4BRA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.102786253 (538h12m24.973869298s) [noise=true , address=45.91.116.14:4189 , piece_id=BSQMVCFMLGQT25XO7HLYNSHOPLFRCW2GN4JETZ6XO4PPGTZRCM3A, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.103507886 (538h12m24.974590921s) [noise=true , address=145.79.227.31:4798 , piece_id=ADBHXG4JO5OTHG4FTEJWMKDOATSBRTDFTX3GR5WLJ3KW36YE6XWA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.103979545 (538h12m24.97506259s) [noise=true , address=145.79.227.7:4293 , piece_id=QOFAONDGKD4SENE3IQNYOYBFFBV6PXXHLLIMGDVGSD735M3552HA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.104328509 (538h12m24.975411554s) [noise=true , address=145.79.227.45:4229 , piece_id=MY5DP2S3NTD74UCKINW547VALGERUGUDVJSWSNIMIRYJFZXUNX7Q, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.104754039 (538h12m24.975837084s) [noise=true , address=45.91.116.7:4012 , piece_id=VFEA4TQKMSRWYA3ONC55MJMIQFYPDS4KWS6FVKC4BYP5ZXHUIGHQ, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.105135976 (538h12m24.976219012s) [noise=true , address=45.91.116.10:4081 , piece_id=UNGVIGJ7KIOVQTDZQEX5NOMT7NWIVQ44M4OBEGARSHQOQQMNFFAA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.105434936 (538h12m24.976517981s) [noise=true , address=145.79.227.13:4803 , piece_id=R4JODTHTC23EDCKZYF5XIKFHNAR6OI7JIRSK6LTX6ZCBC3M5ZJUQ, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.105895222 (538h12m24.976978257s) [noise=true , address=45.91.116.5:4298 , piece_id=3UKKGWB6X5Z6AJON5CP34GGGDAMNW4YFD7OFHH7JZQG4IMKKXVVA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.107103393 (538h12m24.978186428s) [noise=true , address=145.79.227.47:4265 , piece_id=VH33CAK5KM3UE5UPUSH3MXJX4JBP3DOE2TP6AGELFSTMGQSOHCYA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.108664767 (538h12m24.979747802s) [noise=true , address=145.79.227.16:4912 , piece_id=HX5OER5Y3LKL2WD2XOJZOYGV2OADF22DO2P2WJMNNWHOL2HUS23A, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.110811944 (538h12m24.98189498s) [noise=true , address=145.79.227.11:4731 , piece_id=XK3D7KT7IXTQLIEOYTPGK5QO74GXWRSGW73AAZO2HBYEQPGBY7NA, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.112350384 (538h12m24.983433419s) [noise=true , address=145.79.227.4:4187 , piece_id=XHIUMDDI5KFSPEJG35UCZ4GEES7TOW3SOMG5VZTIYVGCGFJWIWVQ, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.113574946 (538h12m24.984657981s) [noise=true , address=145.79.227.46:4602 , piece_id=UQZLC5F5E24G72MSM5HJ4ZXDYKUJ6LIJ6MY46H6MVJSBSFZF5P7Q, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.114784489 (538h12m24.985867524s) [noise=true , address=145.79.227.51:4410 , piece_id=6WZQOYZQIGHLHUTLWPCDG2DAWB3YVG4NLBSUX7NYV2AQHZZIVJGQ, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:25.115976209 (538h12m24.987059244s) [noise=true , address=145.79.227.41:4085 , piece_id=D3IBHA3CZSM63HRVU3JMZXG6QRZDJOWLZXJHYZFVNTDIKOCEYZ7Q, seg_pos={0 0}, upload=472008]: Uploading piece... Dec 28 01:41:26.953200478 (538h12m26.824283524s) [upload=472008]: Next returned nil segment Dec 28 01:41:26.953225837 (538h12m26.824308882s) [upload=472008]: Waiting for error group managing segments... Dec 28 01:41:27.261688682 (538h12m27.132771717s) [noise=true , address=145.79.227.7:4293 , piece_id=QOFAONDGKD4SENE3IQNYOYBFFBV6PXXHLLIMGDVGSD735M3552HA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.266268774 (538h12m27.137351809s) [noise=true , address=145.79.227.46:4602 , piece_id=UQZLC5F5E24G72MSM5HJ4ZXDYKUJ6LIJ6MY46H6MVJSBSFZF5P7Q, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.269348929 (538h12m27.140431974s) [noise=true , address=145.79.227.51:4410 , piece_id=6WZQOYZQIGHLHUTLWPCDG2DAWB3YVG4NLBSUX7NYV2AQHZZIVJGQ, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.272709088 (538h12m27.143792123s) [noise=true , address=145.79.227.41:4085 , piece_id=D3IBHA3CZSM63HRVU3JMZXG6QRZDJOWLZXJHYZFVNTDIKOCEYZ7Q, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.274099906 (538h12m27.145182951s) [noise=true , address=145.79.227.11:4731 , piece_id=XK3D7KT7IXTQLIEOYTPGK5QO74GXWRSGW73AAZO2HBYEQPGBY7NA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.282025988 (538h12m27.153109023s) [noise=true , address=145.79.227.19:4077 , piece_id=GBL3ZP7LUZCS64MOOUOESRXE3TGCEMCSLJY266TZKZ4PPPRG4BRA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.291921161 (538h12m27.163004216s) [noise=true , address=145.79.227.45:4229 , piece_id=MY5DP2S3NTD74UCKINW547VALGERUGUDVJSWSNIMIRYJFZXUNX7Q, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.301076424 (538h12m27.172159459s) [noise=true , address=145.79.227.31:4798 , piece_id=ADBHXG4JO5OTHG4FTEJWMKDOATSBRTDFTX3GR5WLJ3KW36YE6XWA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.306165815 (538h12m27.17724886s) [noise=true , address=145.79.227.47:4265 , piece_id=VH33CAK5KM3UE5UPUSH3MXJX4JBP3DOE2TP6AGELFSTMGQSOHCYA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.314052752 (538h12m27.185135797s) [noise=true , address=45.91.116.7:4012 , piece_id=VFEA4TQKMSRWYA3ONC55MJMIQFYPDS4KWS6FVKC4BYP5ZXHUIGHQ, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.364383922 (538h12m27.235466967s) [noise=true , address=45.91.116.14:4189 , piece_id=BSQMVCFMLGQT25XO7HLYNSHOPLFRCW2GN4JETZ6XO4PPGTZRCM3A, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.372911369 (538h12m27.243994405s) [noise=true , address=45.91.116.10:4081 , piece_id=UNGVIGJ7KIOVQTDZQEX5NOMT7NWIVQ44M4OBEGARSHQOQQMNFFAA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.425290579 (538h12m27.296373614s) [noise=true , address=45.91.116.5:4298 , piece_id=3UKKGWB6X5Z6AJON5CP34GGGDAMNW4YFD7OFHH7JZQG4IMKKXVVA, seg_pos={0 0}, upload=472008]: Done uploading piece. err: <nil> Dec 28 01:41:27.425312972 (538h12m27.296396007s) [seg_pos={0 0}, upload=472008]: Segment reached optimal threshold of 13 pieces. Dec 28 01:41:27.425484589 (538h12m27.296567624s) [noise=true , address=145.79.227.16:4912 , piece_id=HX5OER5Y3LKL2WD2XOJZOYGV2OADF22DO2P2WJMNNWHOL2HUS23A, seg_pos={0 0}, upload=472008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:35720->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:35720->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35720->145.79.227.16:4912: use of closed network connection Dec 28 01:41:27.425490680 (538h12m27.296573715s) [noise=true , address=145.79.227.16:4912 , piece_id=HX5OER5Y3LKL2WD2XOJZOYGV2OADF22DO2P2WJMNNWHOL2HUS23A, seg_pos={0 0}, upload=472008]: Long tail cancellation due to optimal threshold Dec 28 01:41:27.425597564 (538h12m27.296680599s) [noise=true , address=145.79.227.4:4187 , piece_id=XHIUMDDI5KFSPEJG35UCZ4GEES7TOW3SOMG5VZTIYVGCGFJWIWVQ, seg_pos={0 0}, upload=472008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:36670->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:36670->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->145.79.227.4:4187: use of closed network connection Dec 28 01:41:27.425605458 (538h12m27.296688503s) [noise=true , address=145.79.227.4:4187 , piece_id=XHIUMDDI5KFSPEJG35UCZ4GEES7TOW3SOMG5VZTIYVGCGFJWIWVQ, seg_pos={0 0}, upload=472008]: Long tail cancellation due to optimal threshold Dec 28 01:41:27.425716029 (538h12m27.296799064s) [noise=true , address=145.79.227.13:4803 , piece_id=R4JODTHTC23EDCKZYF5XIKFHNAR6OI7JIRSK6LTX6ZCBC3M5ZJUQ, seg_pos={0 0}, upload=472008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:56192->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56192->145.79.227.13:4803: use of closed network connection Dec 28 01:41:27.425721870 (538h12m27.296804905s) [noise=true , address=145.79.227.13:4803 , piece_id=R4JODTHTC23EDCKZYF5XIKFHNAR6OI7JIRSK6LTX6ZCBC3M5ZJUQ, seg_pos={0 0}, upload=472008]: Long tail cancellation due to optimal threshold Dec 28 01:41:27.425747869 (538h12m27.296830905s) [seg_pos={0 0}, upload=472008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:41:27.425836068 (538h12m27.296919103s) [upload=472008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:41:27.425843051 (538h12m27.296926086s) [upload=472008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:41:28.886345198 (538h12m28.757428243s) [upload=472008]: Done uploading segments... Dec 28 01:41:28.886370376 (538h12m28.757453411s) [upload=472008]: Upload finished. err: <nil> Duration: 30.00347853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.X] duration: 30.003313759s trace: https://jaeger.dev.storj.io/trace/44089c675bc499bd FAIL (2025-12-28 01:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 01:40:54.567910282 (538h11m54.438993317s) [upload=471999]: Starting upload Dec 28 01:40:54.567989584 (538h11m54.439072619s) [upload=471999]: Done starting upload Dec 28 01:40:54.568277412 (538h11m54.439360457s) [upload=471999]: Uploading segments... Dec 28 01:40:54.568333177 (538h11m54.439416212s) [upload=471999]: Got next segment. Inline: false Dec 28 01:40:54.568365439 (538h11m54.439448474s) [upload=471999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:40:54.568370830 (538h11m54.439453875s) [upload=471999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:40:54.831175438 (538h11m54.702258483s) [seg_pos={0 0}, upload=471999]: Begin upload segment... Dec 28 01:40:54.831226616 (538h11m54.702309661s) [seg_pos={0 0}, upload=471999]: Done begin upload segment. Dec 28 01:40:54.831258526 (538h11m54.702341561s) [noise=true , address=145.79.227.18:4007 , piece_id=7AYPHVDNSTNFQ5TBFNUYGLZOYFYFERB5XLABWHMW4ZR4S52E5F2A, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.831299033 (538h11m54.702382068s) [noise=true , address=145.79.227.3:4151 , piece_id=WUP73GO2Y7YF3P4DBWSERJTLWY2JSPAFQOYJU73JBIH4UTFHK6JA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.831387202 (538h11m54.702470237s) [noise=true , address=145.79.227.34:4907 , piece_id=WZXAFFG3WPFC7QRKOAD377VYMP4RRCNSZDUUOYIPAMNCRW3XI2SQ, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.831737659 (538h11m54.702820694s) [noise=true , address=45.91.116.11:4117 , piece_id=YSK4W66KZQTX4U2COQ5K6QCBFF253P65PMKNSCCUP7UPYH6KQ3GA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.832026619 (538h11m54.703109664s) [noise=true , address=145.79.227.50:4374 , piece_id=ALCCW4JI42CTFDRUNIKVBBPGGDNNMSGSPRR6G4PLVR242KLWYEIQ, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.832320378 (538h11m54.703403413s) [noise=true , address=145.79.227.11:4730 , piece_id=TWJWBL4MOEUKNILFLE4QPD35EAZLY3MUH5QV7GV4QRABOGMJAVIA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.832599069 (538h11m54.703682104s) [noise=true , address=145.79.227.14:4838 , piece_id=TYESD3F33QMT4HNRD7GEKBAHWQOUCSUSKWCOSR46QOFPTVUSNHUQ, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.832981217 (538h11m54.704064262s) [noise=true , address=45.91.116.6:4973 , piece_id=QZ5BQPQ3ZZRZZLDZVWPY2FVR66PCRPRP23TJBGWE3IBHT6QLILEA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.834571455 (538h11m54.7056545s) [noise=true , address=77.90.31.14:4664 , piece_id=GMZ5UBJ6EKUMTSI74HOZXQ4J7GJM6DSKU6MMRY6UCNZNN4ELXP7Q, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.836178405 (538h11m54.70726145s) [noise=true , address=145.79.227.12:4767 , piece_id=JPJBUDNJTTNG2QUE76FG5A2FJHDNUBWXEFPDQHLLSQHDG3OZQDJA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.837665597 (538h11m54.708748643s) [noise=true , address=145.79.227.13:4804 , piece_id=7LVRMEVFKEG5ZCO2FNOSZMDCVXQSDMVDYZEZLARIVTHAO3NNXFTA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.839434827 (538h11m54.710517862s) [noise=true , address=45.91.116.10:4083 , piece_id=NLRRY4BENSV3KXRY5QU6F4JOQB6SDHDTU2M7XLVPOFP5INC3K6KA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.841153620 (538h11m54.712236665s) [noise=true , address=145.79.227.16:4912 , piece_id=E4JXOTFGHPJRTYDHID3GVEMSH7WTTVUADGM4ST52FRE7CHXUWTWA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.842405845 (538h11m54.71348888s) [noise=true , address=145.79.227.45:4229 , piece_id=BZOXHKJ5M7VNBXWHZOGDEMLNJHQJQJS3VO6NQ7EEAUASEK4ZDHNQ, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.843115395 (538h11m54.71419844s) [noise=true , address=145.79.227.9:4367 , piece_id=TARIMR2N7G7XPQYFEDUJYL5CPL3I6OT7AOCLSSZT6KZWPUTXOD6Q, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:54.844393519 (538h11m54.715476564s) [noise=true , address=77.90.31.16:4735 , piece_id=XVDGHRWD52DPNWUJEAIOC4JABIZFB5QXBG7ZESI2Q6SI5Z5JVZFA, seg_pos={0 0}, upload=471999]: Uploading piece... Dec 28 01:40:56.650284877 (538h11m56.521367912s) [upload=471999]: Next returned nil segment Dec 28 01:40:56.650310236 (538h11m56.521393271s) [upload=471999]: Waiting for error group managing segments... Dec 28 01:40:57.021609946 (538h11m56.892693011s) [noise=true , address=145.79.227.45:4229 , piece_id=BZOXHKJ5M7VNBXWHZOGDEMLNJHQJQJS3VO6NQ7EEAUASEK4ZDHNQ, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.027474604 (538h11m56.898557639s) [noise=true , address=145.79.227.16:4912 , piece_id=E4JXOTFGHPJRTYDHID3GVEMSH7WTTVUADGM4ST52FRE7CHXUWTWA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.086072154 (538h11m56.957155189s) [noise=true , address=77.90.31.16:4735 , piece_id=XVDGHRWD52DPNWUJEAIOC4JABIZFB5QXBG7ZESI2Q6SI5Z5JVZFA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.088980001 (538h11m56.960063046s) [noise=true , address=45.91.116.10:4083 , piece_id=NLRRY4BENSV3KXRY5QU6F4JOQB6SDHDTU2M7XLVPOFP5INC3K6KA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.130222882 (538h11m57.001305927s) [noise=true , address=77.90.31.14:4664 , piece_id=GMZ5UBJ6EKUMTSI74HOZXQ4J7GJM6DSKU6MMRY6UCNZNN4ELXP7Q, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.311927239 (538h11m57.183010274s) [noise=true , address=145.79.227.11:4730 , piece_id=TWJWBL4MOEUKNILFLE4QPD35EAZLY3MUH5QV7GV4QRABOGMJAVIA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.347156056 (538h11m57.218239091s) [noise=true , address=145.79.227.18:4007 , piece_id=7AYPHVDNSTNFQ5TBFNUYGLZOYFYFERB5XLABWHMW4ZR4S52E5F2A, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.506308917 (538h11m57.377391952s) [noise=true , address=145.79.227.14:4838 , piece_id=TYESD3F33QMT4HNRD7GEKBAHWQOUCSUSKWCOSR46QOFPTVUSNHUQ, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.557310101 (538h11m57.428393136s) [noise=true , address=45.91.116.6:4973 , piece_id=QZ5BQPQ3ZZRZZLDZVWPY2FVR66PCRPRP23TJBGWE3IBHT6QLILEA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:57.710948651 (538h11m57.582031697s) [noise=true , address=145.79.227.12:4767 , piece_id=JPJBUDNJTTNG2QUE76FG5A2FJHDNUBWXEFPDQHLLSQHDG3OZQDJA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:58.088114412 (538h11m57.959197447s) [noise=true , address=145.79.227.13:4804 , piece_id=7LVRMEVFKEG5ZCO2FNOSZMDCVXQSDMVDYZEZLARIVTHAO3NNXFTA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:58.101699563 (538h11m57.972782599s) [noise=true , address=145.79.227.9:4367 , piece_id=TARIMR2N7G7XPQYFEDUJYL5CPL3I6OT7AOCLSSZT6KZWPUTXOD6Q, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:58.321111147 (538h11m58.192194182s) [noise=true , address=145.79.227.34:4907 , piece_id=WZXAFFG3WPFC7QRKOAD377VYMP4RRCNSZDUUOYIPAMNCRW3XI2SQ, seg_pos={0 0}, upload=471999]: Done uploading piece. err: <nil> Dec 28 01:40:58.321129271 (538h11m58.192212316s) [seg_pos={0 0}, upload=471999]: Segment reached optimal threshold of 13 pieces. Dec 28 01:40:58.321326637 (538h11m58.192409672s) [noise=true , address=45.91.116.11:4117 , piece_id=YSK4W66KZQTX4U2COQ5K6QCBFF253P65PMKNSCCUP7UPYH6KQ3GA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 01:40:58.321335955 (538h11m58.192419s) [noise=true , address=45.91.116.11:4117 , piece_id=YSK4W66KZQTX4U2COQ5K6QCBFF253P65PMKNSCCUP7UPYH6KQ3GA, seg_pos={0 0}, upload=471999]: Long tail cancellation due to optimal threshold Dec 28 01:40:58.321463307 (538h11m58.192546342s) [noise=true , address=145.79.227.50:4374 , piece_id=ALCCW4JI42CTFDRUNIKVBBPGGDNNMSGSPRR6G4PLVR242KLWYEIQ, seg_pos={0 0}, upload=471999]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:42302->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:42302->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42302->145.79.227.50:4374: use of closed network connection Dec 28 01:40:58.321474278 (538h11m58.192557313s) [noise=true , address=145.79.227.50:4374 , piece_id=ALCCW4JI42CTFDRUNIKVBBPGGDNNMSGSPRR6G4PLVR242KLWYEIQ, seg_pos={0 0}, upload=471999]: Long tail cancellation due to optimal threshold Dec 28 01:40:58.321574138 (538h11m58.192657193s) [noise=true , address=145.79.227.3:4151 , piece_id=WUP73GO2Y7YF3P4DBWSERJTLWY2JSPAFQOYJU73JBIH4UTFHK6JA, seg_pos={0 0}, upload=471999]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 28 01:40:58.321584078 (538h11m58.192667133s) [noise=true , address=145.79.227.3:4151 , piece_id=WUP73GO2Y7YF3P4DBWSERJTLWY2JSPAFQOYJU73JBIH4UTFHK6JA, seg_pos={0 0}, upload=471999]: Long tail cancellation due to optimal threshold Dec 28 01:40:58.321604596 (538h11m58.192687641s) [seg_pos={0 0}, upload=471999]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:40:58.321678928 (538h11m58.192761963s) [upload=471999]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:40:58.321684438 (538h11m58.192767473s) [upload=471999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:40:58.633908988 (538h11m58.504992023s) [upload=471999]: Done uploading segments... Dec 28 01:40:58.633926492 (538h11m58.505009527s) [upload=471999]: Upload finished. err: <nil> Duration: 30.003339256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X.] duration: 6.126358234s trace: https://jaeger.dev.storj.io/trace/62684b9406191c27 FAIL (2025-12-28 01:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:39:20.223655401 (538h10m20.094738436s) [upload=471980]: Starting upload Dec 28 01:39:20.223753599 (538h10m20.094836634s) [upload=471980]: Done starting upload Dec 28 01:39:20.223964230 (538h10m20.095047265s) [upload=471980]: Uploading segments... Dec 28 01:39:20.224871788 (538h10m20.095954823s) [upload=471980]: Got next segment. Inline: false Dec 28 01:39:20.224910581 (538h10m20.095993616s) [upload=471980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:39:20.224915381 (538h10m20.095998427s) [upload=471980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:39:20.474699324 (538h10m20.345782359s) [seg_pos={0 0}, upload=471980]: Begin upload segment... Dec 28 01:39:20.474927579 (538h10m20.346010624s) [seg_pos={0 0}, upload=471980]: Done begin upload segment. Dec 28 01:39:20.474970721 (538h10m20.346053756s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.475023792 (538h10m20.346106837s) [noise=true , address=145.79.227.6:4258 , piece_id=CT54WNJZSI455RM6EMGM4VTXHVQKDACANIURWVF2RGVVCCD4IW7A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.476138505 (538h10m20.34722155s) [noise=true , address=145.79.227.48:4303 , piece_id=6TZRYDSOE67HPCQVPHBE4VAYWUNIXJZ7NSNBM56IE3IEWE5XRNYQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.476848476 (538h10m20.347931521s) [noise=true , address=145.79.227.37:4566 , piece_id=7PNNOMB2HNIJVQNPL4IERVXGPNCTGPFLS2XXM6OQAJVT6S5I7K7Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.477490299 (538h10m20.348573344s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.478192837 (538h10m20.349275882s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.478917957 (538h10m20.350000992s) [noise=true , address=45.91.116.6:4973 , piece_id=F6FISSPMFKR6IEZ7PXUEHV2E2DAELY3SKIL7U2EAWGTSXAOGIP5Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.479487502 (538h10m20.350570547s) [noise=true , address=77.90.31.10:4520 , piece_id=D2JLACLHRIVT5STZOD2IYPMFROQAV6PK5LVRW3N4FRJV2JFVCQLQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.480166265 (538h10m20.35124931s) [noise=true , address=45.91.116.14:4192 , piece_id=UVFIUJJLGSU7PKV2QGYANZTE5ZMQB3D56DIR4YMP3STCKJ4QSI6A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.481973466 (538h10m20.353056501s) [noise=true , address=145.79.227.13:4803 , piece_id=CEZ23U3LRUCJQEBDSV4CW64VXAP6CNF2OY4MHLN57RJK536TEAZA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.483257000 (538h10m20.354340035s) [noise=true , address=77.90.31.14:4663 , piece_id=QRKJAGFH7YRVIKDIHV4LVF6ARND4J7SUF4MH72ZXOGVMK5OQEPZA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.484204255 (538h10m20.35528729s) [noise=true , address=145.79.227.36:4518 , piece_id=QDIK2XSGUM7ICJTBPA5QEKUE4TK7EC2MVXGT4KLVHETVJZOJP2PA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.484897064 (538h10m20.355980099s) [noise=true , address=145.79.227.42:4123 , piece_id=XW3EX5JOMJZVADEFZBMQ7HRAFO2RD4ZFEH3TJ767HOGTT5QOVR7Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.485386506 (538h10m20.356469551s) [noise=true , address=145.79.227.19:4079 , piece_id=CJNG25IRMYFVNZPZ3ZCLIXIOXUE3KM52HP35A2QZXMAGBPUCDXEQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.486144139 (538h10m20.357227174s) [noise=true , address=45.91.116.5:4297 , piece_id=V3336T7RPRYCFSVKMVPLXBRGQKMIWJRGLITLH5DHJD4KIK77JRVA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.486563998 (538h10m20.357647043s) [noise=true , address=145.79.227.16:4910 , piece_id=XI6HADNGMHS7ZDDUXBOJMANV2ZPVMDR2B4ECENP3WFWFJOFQSGKQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:22.205206283 (538h10m22.076289318s) [upload=471980]: Next returned nil segment Dec 28 01:39:22.205219488 (538h10m22.076302523s) [upload=471980]: Waiting for error group managing segments... Dec 28 01:39:22.487638209 (538h10m22.358721254s) [noise=true , address=145.79.227.42:4123 , piece_id=XW3EX5JOMJZVADEFZBMQ7HRAFO2RD4ZFEH3TJ767HOGTT5QOVR7Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.510740474 (538h10m22.381823519s) [noise=true , address=145.79.227.6:4258 , piece_id=CT54WNJZSI455RM6EMGM4VTXHVQKDACANIURWVF2RGVVCCD4IW7A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.511987609 (538h10m22.383070654s) [noise=true , address=145.79.227.19:4079 , piece_id=CJNG25IRMYFVNZPZ3ZCLIXIOXUE3KM52HP35A2QZXMAGBPUCDXEQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.516857171 (538h10m22.387940217s) [noise=true , address=145.79.227.13:4803 , piece_id=CEZ23U3LRUCJQEBDSV4CW64VXAP6CNF2OY4MHLN57RJK536TEAZA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.542741955 (538h10m22.413825s) [noise=true , address=145.79.227.16:4910 , piece_id=XI6HADNGMHS7ZDDUXBOJMANV2ZPVMDR2B4ECENP3WFWFJOFQSGKQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.592916655 (538h10m22.46399969s) [noise=true , address=145.79.227.48:4303 , piece_id=6TZRYDSOE67HPCQVPHBE4VAYWUNIXJZ7NSNBM56IE3IEWE5XRNYQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.597532215 (538h10m22.46861526s) [noise=true , address=145.79.227.36:4518 , piece_id=QDIK2XSGUM7ICJTBPA5QEKUE4TK7EC2MVXGT4KLVHETVJZOJP2PA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.623392762 (538h10m22.494475817s) [noise=true , address=145.79.227.37:4566 , piece_id=7PNNOMB2HNIJVQNPL4IERVXGPNCTGPFLS2XXM6OQAJVT6S5I7K7Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.685597712 (538h10m22.556680757s) [noise=true , address=77.90.31.10:4520 , piece_id=D2JLACLHRIVT5STZOD2IYPMFROQAV6PK5LVRW3N4FRJV2JFVCQLQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.727106238 (538h10m22.598189283s) [noise=true , address=45.91.116.6:4973 , piece_id=F6FISSPMFKR6IEZ7PXUEHV2E2DAELY3SKIL7U2EAWGTSXAOGIP5Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.746434516 (538h10m22.617517561s) [noise=true , address=45.91.116.5:4297 , piece_id=V3336T7RPRYCFSVKMVPLXBRGQKMIWJRGLITLH5DHJD4KIK77JRVA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.769400421 (538h10m22.640483466s) [noise=true , address=45.91.116.14:4192 , piece_id=UVFIUJJLGSU7PKV2QGYANZTE5ZMQB3D56DIR4YMP3STCKJ4QSI6A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.785124905 (538h10m22.65620794s) [noise=true , address=77.90.31.14:4663 , piece_id=QRKJAGFH7YRVIKDIHV4LVF6ARND4J7SUF4MH72ZXOGVMK5OQEPZA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.785145955 (538h10m22.656229001s) [seg_pos={0 0}, upload=471980]: Segment reached optimal threshold of 13 pieces. Dec 28 01:39:22.785380512 (538h10m22.656463567s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785395150 (538h10m22.656478225s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785443532 (538h10m22.656526567s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785454754 (538h10m22.656537799s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785653603 (538h10m22.656736638s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785660956 (538h10m22.656744001s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785678379 (538h10m22.656761434s) [seg_pos={0 0}, upload=471980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:39:22.785775394 (538h10m22.656858439s) [upload=471980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:39:22.785782056 (538h10m22.656865102s) [upload=471980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:39:23.094052799 (538h10m22.965135845s) [upload=471980]: Done uploading segments... Dec 28 01:39:23.094071866 (538h10m22.965154901s) [upload=471980]: Upload finished. err: <nil> Duration: 6.1263872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..X] duration: 30.002645038s trace: https://jaeger.dev.storj.io/trace/62684b9406191c27 FAIL (2025-12-28 01:39:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:39:20.223655401 (538h10m20.094738436s) [upload=471980]: Starting upload Dec 28 01:39:20.223753599 (538h10m20.094836634s) [upload=471980]: Done starting upload Dec 28 01:39:20.223964230 (538h10m20.095047265s) [upload=471980]: Uploading segments... Dec 28 01:39:20.224871788 (538h10m20.095954823s) [upload=471980]: Got next segment. Inline: false Dec 28 01:39:20.224910581 (538h10m20.095993616s) [upload=471980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:39:20.224915381 (538h10m20.095998427s) [upload=471980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:39:20.474699324 (538h10m20.345782359s) [seg_pos={0 0}, upload=471980]: Begin upload segment... Dec 28 01:39:20.474927579 (538h10m20.346010624s) [seg_pos={0 0}, upload=471980]: Done begin upload segment. Dec 28 01:39:20.474970721 (538h10m20.346053756s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.475023792 (538h10m20.346106837s) [noise=true , address=145.79.227.6:4258 , piece_id=CT54WNJZSI455RM6EMGM4VTXHVQKDACANIURWVF2RGVVCCD4IW7A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.476138505 (538h10m20.34722155s) [noise=true , address=145.79.227.48:4303 , piece_id=6TZRYDSOE67HPCQVPHBE4VAYWUNIXJZ7NSNBM56IE3IEWE5XRNYQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.476848476 (538h10m20.347931521s) [noise=true , address=145.79.227.37:4566 , piece_id=7PNNOMB2HNIJVQNPL4IERVXGPNCTGPFLS2XXM6OQAJVT6S5I7K7Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.477490299 (538h10m20.348573344s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.478192837 (538h10m20.349275882s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.478917957 (538h10m20.350000992s) [noise=true , address=45.91.116.6:4973 , piece_id=F6FISSPMFKR6IEZ7PXUEHV2E2DAELY3SKIL7U2EAWGTSXAOGIP5Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.479487502 (538h10m20.350570547s) [noise=true , address=77.90.31.10:4520 , piece_id=D2JLACLHRIVT5STZOD2IYPMFROQAV6PK5LVRW3N4FRJV2JFVCQLQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.480166265 (538h10m20.35124931s) [noise=true , address=45.91.116.14:4192 , piece_id=UVFIUJJLGSU7PKV2QGYANZTE5ZMQB3D56DIR4YMP3STCKJ4QSI6A, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.481973466 (538h10m20.353056501s) [noise=true , address=145.79.227.13:4803 , piece_id=CEZ23U3LRUCJQEBDSV4CW64VXAP6CNF2OY4MHLN57RJK536TEAZA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.483257000 (538h10m20.354340035s) [noise=true , address=77.90.31.14:4663 , piece_id=QRKJAGFH7YRVIKDIHV4LVF6ARND4J7SUF4MH72ZXOGVMK5OQEPZA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.484204255 (538h10m20.35528729s) [noise=true , address=145.79.227.36:4518 , piece_id=QDIK2XSGUM7ICJTBPA5QEKUE4TK7EC2MVXGT4KLVHETVJZOJP2PA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.484897064 (538h10m20.355980099s) [noise=true , address=145.79.227.42:4123 , piece_id=XW3EX5JOMJZVADEFZBMQ7HRAFO2RD4ZFEH3TJ767HOGTT5QOVR7Q, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.485386506 (538h10m20.356469551s) [noise=true , address=145.79.227.19:4079 , piece_id=CJNG25IRMYFVNZPZ3ZCLIXIOXUE3KM52HP35A2QZXMAGBPUCDXEQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.486144139 (538h10m20.357227174s) [noise=true , address=45.91.116.5:4297 , piece_id=V3336T7RPRYCFSVKMVPLXBRGQKMIWJRGLITLH5DHJD4KIK77JRVA, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:20.486563998 (538h10m20.357647043s) [noise=true , address=145.79.227.16:4910 , piece_id=XI6HADNGMHS7ZDDUXBOJMANV2ZPVMDR2B4ECENP3WFWFJOFQSGKQ, seg_pos={0 0}, upload=471980]: Uploading piece... Dec 28 01:39:22.205206283 (538h10m22.076289318s) [upload=471980]: Next returned nil segment Dec 28 01:39:22.205219488 (538h10m22.076302523s) [upload=471980]: Waiting for error group managing segments... Dec 28 01:39:22.487638209 (538h10m22.358721254s) [noise=true , address=145.79.227.42:4123 , piece_id=XW3EX5JOMJZVADEFZBMQ7HRAFO2RD4ZFEH3TJ767HOGTT5QOVR7Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.510740474 (538h10m22.381823519s) [noise=true , address=145.79.227.6:4258 , piece_id=CT54WNJZSI455RM6EMGM4VTXHVQKDACANIURWVF2RGVVCCD4IW7A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.511987609 (538h10m22.383070654s) [noise=true , address=145.79.227.19:4079 , piece_id=CJNG25IRMYFVNZPZ3ZCLIXIOXUE3KM52HP35A2QZXMAGBPUCDXEQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.516857171 (538h10m22.387940217s) [noise=true , address=145.79.227.13:4803 , piece_id=CEZ23U3LRUCJQEBDSV4CW64VXAP6CNF2OY4MHLN57RJK536TEAZA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.542741955 (538h10m22.413825s) [noise=true , address=145.79.227.16:4910 , piece_id=XI6HADNGMHS7ZDDUXBOJMANV2ZPVMDR2B4ECENP3WFWFJOFQSGKQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.592916655 (538h10m22.46399969s) [noise=true , address=145.79.227.48:4303 , piece_id=6TZRYDSOE67HPCQVPHBE4VAYWUNIXJZ7NSNBM56IE3IEWE5XRNYQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.597532215 (538h10m22.46861526s) [noise=true , address=145.79.227.36:4518 , piece_id=QDIK2XSGUM7ICJTBPA5QEKUE4TK7EC2MVXGT4KLVHETVJZOJP2PA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.623392762 (538h10m22.494475817s) [noise=true , address=145.79.227.37:4566 , piece_id=7PNNOMB2HNIJVQNPL4IERVXGPNCTGPFLS2XXM6OQAJVT6S5I7K7Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.685597712 (538h10m22.556680757s) [noise=true , address=77.90.31.10:4520 , piece_id=D2JLACLHRIVT5STZOD2IYPMFROQAV6PK5LVRW3N4FRJV2JFVCQLQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.727106238 (538h10m22.598189283s) [noise=true , address=45.91.116.6:4973 , piece_id=F6FISSPMFKR6IEZ7PXUEHV2E2DAELY3SKIL7U2EAWGTSXAOGIP5Q, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.746434516 (538h10m22.617517561s) [noise=true , address=45.91.116.5:4297 , piece_id=V3336T7RPRYCFSVKMVPLXBRGQKMIWJRGLITLH5DHJD4KIK77JRVA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.769400421 (538h10m22.640483466s) [noise=true , address=45.91.116.14:4192 , piece_id=UVFIUJJLGSU7PKV2QGYANZTE5ZMQB3D56DIR4YMP3STCKJ4QSI6A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.785124905 (538h10m22.65620794s) [noise=true , address=77.90.31.14:4663 , piece_id=QRKJAGFH7YRVIKDIHV4LVF6ARND4J7SUF4MH72ZXOGVMK5OQEPZA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: <nil> Dec 28 01:39:22.785145955 (538h10m22.656229001s) [seg_pos={0 0}, upload=471980]: Segment reached optimal threshold of 13 pieces. Dec 28 01:39:22.785380512 (538h10m22.656463567s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785395150 (538h10m22.656478225s) [noise=true , address=145.79.227.34:4907 , piece_id=TMDYKZMFBZJXXQEXJYMPTOEIOZDMT4VK5VHWJDZWJ34LW77NYYPQ, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785443532 (538h10m22.656526567s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785454754 (538h10m22.656537799s) [noise=true , address=145.79.227.12:4765 , piece_id=2IDS6LBIXJEVEZZTKZHV3EBB4244EXE6CNNERNYPNUJLVBSBA54A, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785653603 (538h10m22.656736638s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 01:39:22.785660956 (538h10m22.656744001s) [noise=true , address=45.91.116.9:4261 , piece_id=5MSRHOBBAX74YR6FQGAN7FO7HAIQXTS4VQL5ZWMVSD5QI5RXNYGA, seg_pos={0 0}, upload=471980]: Long tail cancellation due to optimal threshold Dec 28 01:39:22.785678379 (538h10m22.656761434s) [seg_pos={0 0}, upload=471980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:39:22.785775394 (538h10m22.656858439s) [upload=471980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:39:22.785782056 (538h10m22.656865102s) [upload=471980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:39:23.094052799 (538h10m22.965135845s) [upload=471980]: Done uploading segments... Dec 28 01:39:23.094071866 (538h10m22.965154901s) [upload=471980]: Upload finished. err: <nil> Duration: 30.002678102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX..] duration: 14.445300371s trace: https://jaeger.dev.storj.io/trace/108644bbf0de42ab FAIL (2025-12-28 01:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q) context deadline exceeded ======LOGS====== Dec 28 01:36:08.480346746 (538h7m8.351429791s) [upload=471931]: Starting upload Dec 28 01:36:08.480432569 (538h7m8.351515614s) [upload=471931]: Done starting upload Dec 28 01:36:08.480540484 (538h7m8.351623529s) [upload=471931]: Uploading segments... Dec 28 01:36:08.480766125 (538h7m8.35184917s) [upload=471931]: Got next segment. Inline: false Dec 28 01:36:08.480815208 (538h7m8.351898253s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:36:08.480820958 (538h7m8.351904003s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:36:08.726849720 (538h7m8.597932755s) [seg_pos={0 0}, upload=471931]: Begin upload segment... Dec 28 01:36:08.726896910 (538h7m8.597979945s) [seg_pos={0 0}, upload=471931]: Done begin upload segment. Dec 28 01:36:08.726922690 (538h7m8.598005735s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.727005728 (538h7m8.598088763s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.728487259 (538h7m8.599570294s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729135613 (538h7m8.600218658s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729856937 (538h7m8.600939972s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730529859 (538h7m8.601612904s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730685094 (538h7m8.601768129s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731781332 (538h7m8.602864377s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731920056 (538h7m8.603003091s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.732901384 (538h7m8.60398442s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733164977 (538h7m8.604248012s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733419291 (538h7m8.604502326s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733974097 (538h7m8.605057142s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734476705 (538h7m8.60555975s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734708106 (538h7m8.605791141s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.735245549 (538h7m8.606328584s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:10.459424308 (538h7m10.330507343s) [upload=471931]: Next returned nil segment Dec 28 01:36:10.459441010 (538h7m10.330524045s) [upload=471931]: Waiting for error group managing segments... Dec 28 01:36:10.740048752 (538h7m10.611131787s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.750969508 (538h7m10.622052543s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.770387086 (538h7m10.641470121s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.779432822 (538h7m10.650515867s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.782159815 (538h7m10.65324286s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.787672172 (538h7m10.658755207s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.824234219 (538h7m10.695317264s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.829420626 (538h7m10.700503661s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.917758381 (538h7m10.788841416s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.958746806 (538h7m10.829829851s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.010432343 (538h7m10.881515388s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.015623679 (538h7m10.886706714s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046858541 (538h7m10.917941576s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046888418 (538h7m10.917971463s) [seg_pos={0 0}, upload=471931]: Segment reached optimal threshold of 13 pieces. Dec 28 01:36:11.047157420 (538h7m10.918240465s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047166838 (538h7m10.918249883s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047226892 (538h7m10.918309927s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047237743 (538h7m10.918320788s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047454646 (538h7m10.918537701s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection Dec 28 01:36:11.047464114 (538h7m10.918547149s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047500653 (538h7m10.918583698s) [seg_pos={0 0}, upload=471931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:36:11.047585155 (538h7m10.9186682s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:36:11.047592108 (538h7m10.918675143s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:36:11.338970490 (538h7m11.210053525s) [upload=471931]: Done uploading segments... Dec 28 01:36:11.338996720 (538h7m11.210079765s) [upload=471931]: Upload finished. err: <nil> Duration: 14.445328936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX.] duration: 6.521768241s trace: https://jaeger.dev.storj.io/trace/108644bbf0de42ab FAIL (2025-12-28 01:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q) context deadline exceeded ======LOGS====== Dec 28 01:36:08.480346746 (538h7m8.351429791s) [upload=471931]: Starting upload Dec 28 01:36:08.480432569 (538h7m8.351515614s) [upload=471931]: Done starting upload Dec 28 01:36:08.480540484 (538h7m8.351623529s) [upload=471931]: Uploading segments... Dec 28 01:36:08.480766125 (538h7m8.35184917s) [upload=471931]: Got next segment. Inline: false Dec 28 01:36:08.480815208 (538h7m8.351898253s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:36:08.480820958 (538h7m8.351904003s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:36:08.726849720 (538h7m8.597932755s) [seg_pos={0 0}, upload=471931]: Begin upload segment... Dec 28 01:36:08.726896910 (538h7m8.597979945s) [seg_pos={0 0}, upload=471931]: Done begin upload segment. Dec 28 01:36:08.726922690 (538h7m8.598005735s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.727005728 (538h7m8.598088763s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.728487259 (538h7m8.599570294s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729135613 (538h7m8.600218658s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729856937 (538h7m8.600939972s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730529859 (538h7m8.601612904s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730685094 (538h7m8.601768129s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731781332 (538h7m8.602864377s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731920056 (538h7m8.603003091s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.732901384 (538h7m8.60398442s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733164977 (538h7m8.604248012s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733419291 (538h7m8.604502326s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733974097 (538h7m8.605057142s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734476705 (538h7m8.60555975s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734708106 (538h7m8.605791141s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.735245549 (538h7m8.606328584s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:10.459424308 (538h7m10.330507343s) [upload=471931]: Next returned nil segment Dec 28 01:36:10.459441010 (538h7m10.330524045s) [upload=471931]: Waiting for error group managing segments... Dec 28 01:36:10.740048752 (538h7m10.611131787s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.750969508 (538h7m10.622052543s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.770387086 (538h7m10.641470121s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.779432822 (538h7m10.650515867s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.782159815 (538h7m10.65324286s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.787672172 (538h7m10.658755207s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.824234219 (538h7m10.695317264s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.829420626 (538h7m10.700503661s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.917758381 (538h7m10.788841416s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.958746806 (538h7m10.829829851s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.010432343 (538h7m10.881515388s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.015623679 (538h7m10.886706714s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046858541 (538h7m10.917941576s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046888418 (538h7m10.917971463s) [seg_pos={0 0}, upload=471931]: Segment reached optimal threshold of 13 pieces. Dec 28 01:36:11.047157420 (538h7m10.918240465s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047166838 (538h7m10.918249883s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047226892 (538h7m10.918309927s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047237743 (538h7m10.918320788s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047454646 (538h7m10.918537701s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection Dec 28 01:36:11.047464114 (538h7m10.918547149s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047500653 (538h7m10.918583698s) [seg_pos={0 0}, upload=471931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:36:11.047585155 (538h7m10.9186682s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:36:11.047592108 (538h7m10.918675143s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:36:11.338970490 (538h7m11.210053525s) [upload=471931]: Done uploading segments... Dec 28 01:36:11.338996720 (538h7m11.210079765s) [upload=471931]: Upload finished. err: <nil> Duration: 6.521794591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXXX] duration: 30.002571507s trace: https://jaeger.dev.storj.io/trace/108644bbf0de42ab FAIL (2025-12-28 01:36:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q) context deadline exceeded ======LOGS====== Dec 28 01:36:08.480346746 (538h7m8.351429791s) [upload=471931]: Starting upload Dec 28 01:36:08.480432569 (538h7m8.351515614s) [upload=471931]: Done starting upload Dec 28 01:36:08.480540484 (538h7m8.351623529s) [upload=471931]: Uploading segments... Dec 28 01:36:08.480766125 (538h7m8.35184917s) [upload=471931]: Got next segment. Inline: false Dec 28 01:36:08.480815208 (538h7m8.351898253s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:36:08.480820958 (538h7m8.351904003s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:36:08.726849720 (538h7m8.597932755s) [seg_pos={0 0}, upload=471931]: Begin upload segment... Dec 28 01:36:08.726896910 (538h7m8.597979945s) [seg_pos={0 0}, upload=471931]: Done begin upload segment. Dec 28 01:36:08.726922690 (538h7m8.598005735s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.727005728 (538h7m8.598088763s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.728487259 (538h7m8.599570294s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729135613 (538h7m8.600218658s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.729856937 (538h7m8.600939972s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730529859 (538h7m8.601612904s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.730685094 (538h7m8.601768129s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731781332 (538h7m8.602864377s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.731920056 (538h7m8.603003091s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.732901384 (538h7m8.60398442s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733164977 (538h7m8.604248012s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733419291 (538h7m8.604502326s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.733974097 (538h7m8.605057142s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734476705 (538h7m8.60555975s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.734708106 (538h7m8.605791141s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:08.735245549 (538h7m8.606328584s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Uploading piece... Dec 28 01:36:10.459424308 (538h7m10.330507343s) [upload=471931]: Next returned nil segment Dec 28 01:36:10.459441010 (538h7m10.330524045s) [upload=471931]: Waiting for error group managing segments... Dec 28 01:36:10.740048752 (538h7m10.611131787s) [noise=true , address=145.79.227.45:4230 , piece_id=RS5E6BI2N2DYI55V4UP6HBE6ASX6BTCULLYQRUYHPWTLCPDKNAYA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.750969508 (538h7m10.622052543s) [noise=true , address=145.79.227.42:4121 , piece_id=5L45FH7T7FECFEL7QLYPIPNCV57XJ3JSRH5A5MH24PVTDDC5W3MA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.770387086 (538h7m10.641470121s) [noise=true , address=145.79.227.39:4015 , piece_id=YEFZRFQZQ3QED2NF44NQCT4OZ42XRPQEFYUTRUD4WH5IPOPDDXFQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.779432822 (538h7m10.650515867s) [noise=true , address=145.79.227.7:4295 , piece_id=XQ53CEJR3QUGKUK25QQ72VK5Q2ZQ2ZUVCW4VAIHACZ4IWZRJ44WQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.782159815 (538h7m10.65324286s) [noise=true , address=145.79.227.15:4875 , piece_id=3Q4AKXZ7H3IT2O74EQOQOSP3BTTJ233N37Q3NM6YPFPBCS326ZSA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.787672172 (538h7m10.658755207s) [noise=true , address=145.79.227.17:4971 , piece_id=SZYJDOQ62SFPFLSBALWNCDO5STCCYKREDZ6WOVT4VFLAHIFKSC7Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.824234219 (538h7m10.695317264s) [noise=true , address=145.79.227.11:4731 , piece_id=OST7KAYPJAOC5R37UQA4A54R7ZZYFH7EQJW75EV3LVZ65AYNQD4Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.829420626 (538h7m10.700503661s) [noise=true , address=145.79.227.46:4601 , piece_id=SNXKQSCYDIFJXXC4OKUUSVLMJEXFYFQAMAFIHSJPE7PM5DXPRFEQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.917758381 (538h7m10.788841416s) [noise=true , address=77.90.31.12:4589 , piece_id=4UH2II3G5SE6TLX43ZND3G3AVYEJZWLUJOM43JRTXL6Y5O6Z7IPQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:10.958746806 (538h7m10.829829851s) [noise=true , address=45.91.116.6:4973 , piece_id=3K3YIBGRARFPXEULP62SFHMKMMMU2KO7373THQSRTJMWFKMFSJ6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.010432343 (538h7m10.881515388s) [noise=true , address=77.90.31.17:4773 , piece_id=WC5GRW6TPRD7LZXCA7SBGGS6FKKRCYBIVN6DTF4R724SDPQZ6JZA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.015623679 (538h7m10.886706714s) [noise=true , address=77.90.31.13:4629 , piece_id=VPCZXNDF7KRPEEQTCXFFO2HBTUXPQRDFJBIEVJ3LXAD4SSF7CWXA, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046858541 (538h7m10.917941576s) [noise=true , address=77.90.31.21:4878 , piece_id=QSL3UV6FZX66VCABFZESRWA542WSEV7UTXV7WIWJMSL52KBUHB6Q, seg_pos={0 0}, upload=471931]: Done uploading piece. err: <nil> Dec 28 01:36:11.046888418 (538h7m10.917971463s) [seg_pos={0 0}, upload=471931]: Segment reached optimal threshold of 13 pieces. Dec 28 01:36:11.047157420 (538h7m10.918240465s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047166838 (538h7m10.918249883s) [noise=true , address=45.91.116.9:4263 , piece_id=RJ2KST3I6RWAPESKPHZDQIYL374AJH6DKU5YYNI2JDJS3SXD6LKQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047226892 (538h7m10.918309927s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 01:36:11.047237743 (538h7m10.918320788s) [noise=true , address=77.90.31.15:4697 , piece_id=MQ7QMDKQBS4NQ3OXEKRAAWAGSUHAN3PVPOVXYY4EYZPZD4E6H2CQ, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047454646 (538h7m10.918537701s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46050->145.79.227.16:4909: use of closed network connection Dec 28 01:36:11.047464114 (538h7m10.918547149s) [noise=true , address=145.79.227.16:4909 , piece_id=MWJMFTIN2PGPVMVWPAYIAFQT3E2YFBLJ4JDX72S6ZY5XR23K7W5A, seg_pos={0 0}, upload=471931]: Long tail cancellation due to optimal threshold Dec 28 01:36:11.047500653 (538h7m10.918583698s) [seg_pos={0 0}, upload=471931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:36:11.047585155 (538h7m10.9186682s) [upload=471931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:36:11.047592108 (538h7m10.918675143s) [upload=471931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:36:11.338970490 (538h7m11.210053525s) [upload=471931]: Done uploading segments... Dec 28 01:36:11.338996720 (538h7m11.210079765s) [upload=471931]: Upload finished. err: <nil> Duration: 30.002582639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXXX] duration: 30.002763167s trace: https://jaeger.dev.storj.io/trace/2b1894bab8ccc930 FAIL (2025-12-28 01:36:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: BQ7OCWNGYMUM4LOK4RARETYQWY5QEA6PY6ZVIXFZDZCKYLHAGPIA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: YQVHVK4TEPNVCSV5TYOTCG5YKVDYLR5A5KP5WYPVIQI5MKA2UQUA) context deadline exceeded ======LOGS====== Dec 28 01:35:38.236041370 (538h6m38.107124405s) [upload=471922]: Starting upload Dec 28 01:35:38.236135149 (538h6m38.107218184s) [upload=471922]: Done starting upload Dec 28 01:35:38.236223396 (538h6m38.107306431s) [upload=471922]: Uploading segments... Dec 28 01:35:38.236489363 (538h6m38.107572408s) [upload=471922]: Got next segment. Inline: false Dec 28 01:35:38.236551031 (538h6m38.107634076s) [upload=471922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:35:38.236584825 (538h6m38.10766786s) [upload=471922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:35:38.482747822 (538h6m38.353830867s) [seg_pos={0 0}, upload=471922]: Begin upload segment... Dec 28 01:35:38.482821883 (538h6m38.353904928s) [seg_pos={0 0}, upload=471922]: Done begin upload segment. Dec 28 01:35:38.482860296 (538h6m38.353943341s) [noise=true , address=45.91.116.13:4155 , piece_id=COLP57OQXK2MLUH2UBTESK6YKS2KVAW42JIQA3SEUNP6DS27PDSQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.482908288 (538h6m38.353991323s) [noise=true , address=145.79.227.38:4979 , piece_id=24DZA2SJZ7S5EPHDDL5RHLYSBFX4CJ3ISEBWO3BVEZ4DPLU6SBEQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.483948098 (538h6m38.355031144s) [noise=true , address=145.79.227.32:4836 , piece_id=HVVXL5BAJOTXQMGDZGHLHSIZS7RKMXN5QI6OZ2UBWD2HBLC5CCNQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.484400430 (538h6m38.355483475s) [noise=true , address=145.79.227.15:4875 , piece_id=OSRNOKVM3NEFEZW2WDQPG5HXS3UQJJ3MA6QOGUBKYGYIRL445IWA, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.484627341 (538h6m38.355710386s) [noise=true , address=145.79.227.29:4941 , piece_id=NMPNZVMLTTQ7OGFIMTFSDBI26AQX76FA4ILGJQIQTBUD6STUKYPQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.484834025 (538h6m38.35591707s) [noise=true , address=145.79.227.40:4050 , piece_id=ADXWE7CC3W4SLBKWYKCVHMSNTFT2EFXCNVDGMMGFOPTIWH5QZLKA, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.485278552 (538h6m38.356361597s) [noise=true , address=145.79.227.9:4367 , piece_id=ZMOB5HL5WDLTLSXPIMJDNACJZLUPWUZHFWT67TV55AHGB7LPJNHQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.485497489 (538h6m38.356580534s) [noise=true , address=45.91.116.9:4262 , piece_id=CK2G3WKCNBVTZS4XTUNR67ZFZFF4YSHFKUMOAERDSP2HXS5AVE4Q, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.485701869 (538h6m38.356784914s) [noise=true , address=145.79.227.46:4603 , piece_id=LUPMAGGE6NPVW3MXA37YK7BHDMERBLSJUWUAEHPYPGG2CDPWGLIA, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.486150642 (538h6m38.357233687s) [noise=true , address=77.90.31.12:4590 , piece_id=BQ7OCWNGYMUM4LOK4RARETYQWY5QEA6PY6ZVIXFZDZCKYLHAGPIA, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.486697333 (538h6m38.357780379s) [noise=true , address=77.90.31.16:4736 , piece_id=JKDKBBCKUUCFNZXCXUMMO5Q3Q24PVZ5Q4DD2G4HWAZKKKELN5JEQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.487137472 (538h6m38.358220517s) [noise=true , address=145.79.227.37:4566 , piece_id=ORKKONAXVVSMSK6OSHDGYVZ5S3ZR6U4GGTO66RFKPMQEX5SRFADQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.487624269 (538h6m38.358707314s) [noise=true , address=145.79.227.13:4803 , piece_id=UGKOMHX6R4632EPHYFXLRTUNR2LBNX4UAZRRVNL24BVQ4Q4KXCNQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.488100706 (538h6m38.359183752s) [noise=true , address=145.79.227.2:4114 , piece_id=PQW4OI7F5O5AK7NH6EM2VJYITFRA4S5QLOIVCDCKB4XXBFSFAJVQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.488552076 (538h6m38.359635121s) [noise=true , address=77.90.31.10:4518 , piece_id=YQVHVK4TEPNVCSV5TYOTCG5YKVDYLR5A5KP5WYPVIQI5MKA2UQUA, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:38.489160775 (538h6m38.36024382s) [noise=true , address=145.79.227.4:4185 , piece_id=VXC3EU2HJDD2SFD7XZUJKIMLPII3EKXBJB6VU7OW2DYUZT2U77GQ, seg_pos={0 0}, upload=471922]: Uploading piece... Dec 28 01:35:40.217206738 (538h6m40.088289773s) [upload=471922]: Next returned nil segment Dec 28 01:35:40.217225052 (538h6m40.088308087s) [upload=471922]: Waiting for error group managing segments... Dec 28 01:35:40.505207619 (538h6m40.376290654s) [noise=true , address=145.79.227.4:4185 , piece_id=VXC3EU2HJDD2SFD7XZUJKIMLPII3EKXBJB6VU7OW2DYUZT2U77GQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.505978306 (538h6m40.377061341s) [noise=true , address=145.79.227.9:4367 , piece_id=ZMOB5HL5WDLTLSXPIMJDNACJZLUPWUZHFWT67TV55AHGB7LPJNHQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.508675723 (538h6m40.379758768s) [noise=true , address=145.79.227.38:4979 , piece_id=24DZA2SJZ7S5EPHDDL5RHLYSBFX4CJ3ISEBWO3BVEZ4DPLU6SBEQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.510150943 (538h6m40.381233988s) [noise=true , address=145.79.227.15:4875 , piece_id=OSRNOKVM3NEFEZW2WDQPG5HXS3UQJJ3MA6QOGUBKYGYIRL445IWA, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.522177174 (538h6m40.393260219s) [noise=true , address=145.79.227.37:4566 , piece_id=ORKKONAXVVSMSK6OSHDGYVZ5S3ZR6U4GGTO66RFKPMQEX5SRFADQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.533054248 (538h6m40.404137284s) [noise=true , address=145.79.227.13:4803 , piece_id=UGKOMHX6R4632EPHYFXLRTUNR2LBNX4UAZRRVNL24BVQ4Q4KXCNQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.557167367 (538h6m40.428250412s) [noise=true , address=145.79.227.32:4836 , piece_id=HVVXL5BAJOTXQMGDZGHLHSIZS7RKMXN5QI6OZ2UBWD2HBLC5CCNQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.585247141 (538h6m40.456330176s) [noise=true , address=145.79.227.29:4941 , piece_id=NMPNZVMLTTQ7OGFIMTFSDBI26AQX76FA4ILGJQIQTBUD6STUKYPQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.586142376 (538h6m40.457225421s) [noise=true , address=145.79.227.2:4114 , piece_id=PQW4OI7F5O5AK7NH6EM2VJYITFRA4S5QLOIVCDCKB4XXBFSFAJVQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.593598966 (538h6m40.464682001s) [noise=true , address=145.79.227.46:4603 , piece_id=LUPMAGGE6NPVW3MXA37YK7BHDMERBLSJUWUAEHPYPGG2CDPWGLIA, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.755854726 (538h6m40.626937761s) [noise=true , address=45.91.116.13:4155 , piece_id=COLP57OQXK2MLUH2UBTESK6YKS2KVAW42JIQA3SEUNP6DS27PDSQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.758122614 (538h6m40.629205659s) [noise=true , address=77.90.31.10:4518 , piece_id=YQVHVK4TEPNVCSV5TYOTCG5YKVDYLR5A5KP5WYPVIQI5MKA2UQUA, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.789837310 (538h6m40.660920345s) [noise=true , address=77.90.31.12:4590 , piece_id=BQ7OCWNGYMUM4LOK4RARETYQWY5QEA6PY6ZVIXFZDZCKYLHAGPIA, seg_pos={0 0}, upload=471922]: Done uploading piece. err: <nil> Dec 28 01:35:40.789855655 (538h6m40.66093869s) [seg_pos={0 0}, upload=471922]: Segment reached optimal threshold of 13 pieces. Dec 28 01:35:40.790039205 (538h6m40.66112224s) [noise=true , address=45.91.116.9:4262 , piece_id=CK2G3WKCNBVTZS4XTUNR67ZFZFF4YSHFKUMOAERDSP2HXS5AVE4Q, seg_pos={0 0}, upload=471922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 01:35:40.790045416 (538h6m40.661128451s) [noise=true , address=45.91.116.9:4262 , piece_id=CK2G3WKCNBVTZS4XTUNR67ZFZFF4YSHFKUMOAERDSP2HXS5AVE4Q, seg_pos={0 0}, upload=471922]: Long tail cancellation due to optimal threshold Dec 28 01:35:40.790181536 (538h6m40.661264571s) [noise=true , address=145.79.227.40:4050 , piece_id=ADXWE7CC3W4SLBKWYKCVHMSNTFT2EFXCNVDGMMGFOPTIWH5QZLKA, seg_pos={0 0}, upload=471922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 01:35:40.790188098 (538h6m40.661271133s) [noise=true , address=145.79.227.40:4050 , piece_id=ADXWE7CC3W4SLBKWYKCVHMSNTFT2EFXCNVDGMMGFOPTIWH5QZLKA, seg_pos={0 0}, upload=471922]: Long tail cancellation due to optimal threshold Dec 28 01:35:40.790229146 (538h6m40.661312181s) [noise=true , address=77.90.31.16:4736 , piece_id=JKDKBBCKUUCFNZXCXUMMO5Q3Q24PVZ5Q4DD2G4HWAZKKKELN5JEQ, seg_pos={0 0}, upload=471922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:37230->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 01:35:40.790245989 (538h6m40.661329024s) [noise=true , address=77.90.31.16:4736 , piece_id=JKDKBBCKUUCFNZXCXUMMO5Q3Q24PVZ5Q4DD2G4HWAZKKKELN5JEQ, seg_pos={0 0}, upload=471922]: Long tail cancellation due to optimal threshold Dec 28 01:35:40.790267359 (538h6m40.661350394s) [seg_pos={0 0}, upload=471922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:35:40.790344385 (538h6m40.66142743s) [upload=471922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:35:40.790349324 (538h6m40.661432359s) [upload=471922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:35:41.078169932 (538h6m40.949252977s) [upload=471922]: Done uploading segments... Dec 28 01:35:41.078193648 (538h6m40.949276683s) [upload=471922]: Upload finished. err: <nil> Duration: 30.002782884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XXX] duration: 30.003244309s trace: https://jaeger.dev.storj.io/trace/74aba8fee903d1d FAIL (2025-12-28 01:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 538h5m44.587152051s) [upload=471911]: Uploading segments... Dec 28 01:34:44.716366693 (538h5m44.587449748s) [upload=471911]: Got next segment. Inline: false Dec 28 01:34:44.716408723 (538h5m44.587491768s) [upload=471911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:34:44.716415736 (538h5m44.587498781s) [upload=471911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:34:44.958719334 (538h5m44.829802399s) [seg_pos={0 0}, upload=471911]: Begin upload segment... Dec 28 01:34:44.958811129 (538h5m44.829894174s) [seg_pos={0 0}, upload=471911]: Done begin upload segment. Dec 28 01:34:44.958828211 (538h5m44.829911246s) [noise=true , address=77.90.31.15:4697 , piece_id=CRK4OZVSF5KGVO6YYPZGZM5AVHBUXXM7Q5IULK6M4HCZHHZP5VMQ, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.958848320 (538h5m44.829931355s) [noise=true , address=145.79.227.11:4731 , piece_id=54JFNSXV5ZNCC46L3VCXYATJXFKIUPRNQ5ZOI5EI2TEBFTBFFVXQ, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.959524707 (538h5m44.830607752s) [noise=true , address=77.90.31.18:4808 , piece_id=Z22V4GXBGEOQY6N6QMT26MFJ6JHNTW7KP355SK2DSLUIDZYF6DAA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.959936973 (538h5m44.831020008s) [noise=true , address=145.79.227.49:4338 , piece_id=CJSFOWK7BAYEKZORBCVU46SDY2HDTS5CN37UR7HIBDJ2DYVBU2WA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.960166811 (538h5m44.831249856s) [noise=true , address=145.79.227.19:4079 , piece_id=A6EGXKIZ47L35QZBQF4POOXTGZVU6AEQIM4EQPKEX7ULLG4RBIIQ, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.960298701 (538h5m44.831381736s) [noise=true , address=45.91.116.11:4119 , piece_id=ZWNPNFXIDG5MGU4NWTC2EF7WFE5Z7LDRR6IWLLCSMHX7VVCXXV5A, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.960539510 (538h5m44.831622545s) [noise=true , address=145.79.227.12:4767 , piece_id=H3R7NAPU4IT2IZ4WFU4KOYRA24CGS2WFTRVGDG6UVGVNEXJHIVCA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.960624572 (538h5m44.831707617s) [noise=true , address=45.91.116.7:4010 , piece_id=3OTYKS4AE2VSJSANYAWTUNUDT65OT5R547A7RFFKJ2A7M6QJ5JYA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.960861934 (538h5m44.831944969s) [noise=true , address=145.79.227.18:4006 , piece_id=GPEON6CFSB6QXM3JYFZND52SGN6G6GWH2DESC5BZDMJOIP3V3L3A, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.961231918 (538h5m44.832314963s) [noise=true , address=145.79.227.16:4912 , piece_id=LTMQGYC3INWG7GRWDJR2WJ6GUA7T46TPORFPOU22JUWQWBQXDYQQ, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.961626300 (538h5m44.832709335s) [noise=true , address=145.79.227.5:4222 , piece_id=5C65MT47UPODOIGDCGBJTGJET22X3OITV6VCOAVXSOTXR5KPJU4A, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.962000973 (538h5m44.833084009s) [noise=true , address=45.91.116.4:4937 , piece_id=QNEAAK74F2Z7A4KR746BSXZYUD2TL6YXLZWNDNXZA4A2RSNXLIMA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.962300173 (538h5m44.833383208s) [noise=true , address=145.79.227.45:4231 , piece_id=RKZFGYSK6GW6LVJJ3N4XR4R3GLO6ZGLRDO3NUJI54IJPSD2Q7PXA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.962414862 (538h5m44.833497897s) [noise=true , address=145.79.227.42:4123 , piece_id=KDX3WZYVS3DGC4GRTEVRFQZY24JC737M3IGNCZ7D5CEQFTMCQEMQ, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.963638602 (538h5m44.834721647s) [noise=true , address=45.91.116.5:4300 , piece_id=34RN4SVFICT7EC6NDG33E43EDNT2D3D4G3RNHOVXYTYIGF4VRD4Q, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:44.965010775 (538h5m44.83609381s) [noise=true , address=45.91.116.14:4190 , piece_id=TOP7ABWKXN2ZOI7Q35TL7EFYJ6YWIZ3B3DSNWUGH7DX3XGVPNUQA, seg_pos={0 0}, upload=471911]: Uploading piece... Dec 28 01:34:46.667067586 (538h5m46.538150621s) [upload=471911]: Next returned nil segment Dec 28 01:34:46.667081523 (538h5m46.538164568s) [upload=471911]: Waiting for error group managing segments... Dec 28 01:34:46.958932867 (538h5m46.830015912s) [noise=true , address=145.79.227.18:4006 , piece_id=GPEON6CFSB6QXM3JYFZND52SGN6G6GWH2DESC5BZDMJOIP3V3L3A, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.965279522 (538h5m46.836362567s) [noise=true , address=145.79.227.11:4731 , piece_id=54JFNSXV5ZNCC46L3VCXYATJXFKIUPRNQ5ZOI5EI2TEBFTBFFVXQ, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.967692798 (538h5m46.838775833s) [noise=true , address=145.79.227.12:4767 , piece_id=H3R7NAPU4IT2IZ4WFU4KOYRA24CGS2WFTRVGDG6UVGVNEXJHIVCA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.972222404 (538h5m46.843305439s) [noise=true , address=145.79.227.45:4231 , piece_id=RKZFGYSK6GW6LVJJ3N4XR4R3GLO6ZGLRDO3NUJI54IJPSD2Q7PXA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.973325355 (538h5m46.8444084s) [noise=true , address=145.79.227.42:4123 , piece_id=KDX3WZYVS3DGC4GRTEVRFQZY24JC737M3IGNCZ7D5CEQFTMCQEMQ, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.990771939 (538h5m46.861854974s) [noise=true , address=145.79.227.19:4079 , piece_id=A6EGXKIZ47L35QZBQF4POOXTGZVU6AEQIM4EQPKEX7ULLG4RBIIQ, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:46.997497947 (538h5m46.868580992s) [noise=true , address=145.79.227.16:4912 , piece_id=LTMQGYC3INWG7GRWDJR2WJ6GUA7T46TPORFPOU22JUWQWBQXDYQQ, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.002545660 (538h5m46.873628705s) [noise=true , address=145.79.227.49:4338 , piece_id=CJSFOWK7BAYEKZORBCVU46SDY2HDTS5CN37UR7HIBDJ2DYVBU2WA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.147235028 (538h5m47.018318063s) [noise=true , address=145.79.227.5:4222 , piece_id=5C65MT47UPODOIGDCGBJTGJET22X3OITV6VCOAVXSOTXR5KPJU4A, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.150493172 (538h5m47.021576218s) [noise=true , address=77.90.31.18:4808 , piece_id=Z22V4GXBGEOQY6N6QMT26MFJ6JHNTW7KP355SK2DSLUIDZYF6DAA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.224228709 (538h5m47.095311755s) [noise=true , address=45.91.116.11:4119 , piece_id=ZWNPNFXIDG5MGU4NWTC2EF7WFE5Z7LDRR6IWLLCSMHX7VVCXXV5A, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.349100157 (538h5m47.220183192s) [noise=true , address=45.91.116.7:4010 , piece_id=3OTYKS4AE2VSJSANYAWTUNUDT65OT5R547A7RFFKJ2A7M6QJ5JYA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.415579937 (538h5m47.286662972s) [noise=true , address=45.91.116.14:4190 , piece_id=TOP7ABWKXN2ZOI7Q35TL7EFYJ6YWIZ3B3DSNWUGH7DX3XGVPNUQA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: <nil> Dec 28 01:34:47.415600085 (538h5m47.28668312s) [seg_pos={0 0}, upload=471911]: Segment reached optimal threshold of 13 pieces. Dec 28 01:34:47.415788965 (538h5m47.28687201s) [noise=true , address=45.91.116.5:4300 , piece_id=34RN4SVFICT7EC6NDG33E43EDNT2D3D4G3RNHOVXYTYIGF4VRD4Q, seg_pos={0 0}, upload=471911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53742->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53742->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53742->45.91.116.5:4300: use of closed network connection Dec 28 01:34:47.415795638 (538h5m47.286878673s) [noise=true , address=45.91.116.5:4300 , piece_id=34RN4SVFICT7EC6NDG33E43EDNT2D3D4G3RNHOVXYTYIGF4VRD4Q, seg_pos={0 0}, upload=471911]: Long tail cancellation due to optimal threshold Dec 28 01:34:47.416043750 (538h5m47.287126785s) [noise=true , address=77.90.31.15:4697 , piece_id=CRK4OZVSF5KGVO6YYPZGZM5AVHBUXXM7Q5IULK6M4HCZHHZP5VMQ, seg_pos={0 0}, upload=471911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:39578->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.15:4697: use of closed network connection Dec 28 01:34:47.416051384 (538h5m47.287134419s) [noise=true , address=77.90.31.15:4697 , piece_id=CRK4OZVSF5KGVO6YYPZGZM5AVHBUXXM7Q5IULK6M4HCZHHZP5VMQ, seg_pos={0 0}, upload=471911]: Long tail cancellation due to optimal threshold Dec 28 01:34:47.416252307 (538h5m47.287335342s) [noise=true , address=45.91.116.4:4937 , piece_id=QNEAAK74F2Z7A4KR746BSXZYUD2TL6YXLZWNDNXZA4A2RSNXLIMA, seg_pos={0 0}, upload=471911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 01:34:47.416258799 (538h5m47.287341834s) [noise=true , address=45.91.116.4:4937 , piece_id=QNEAAK74F2Z7A4KR746BSXZYUD2TL6YXLZWNDNXZA4A2RSNXLIMA, seg_pos={0 0}, upload=471911]: Long tail cancellation due to optimal threshold Dec 28 01:34:47.416275081 (538h5m47.287358126s) [seg_pos={0 0}, upload=471911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:34:47.416341407 (538h5m47.287424442s) [upload=471911]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:34:47.416346607 (538h5m47.287429642s) [upload=471911]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:34:47.720564852 (538h5m47.591647907s) [upload=471911]: Done uploading segments... Dec 28 01:34:47.720589008 (538h5m47.591672043s) [upload=471911]: Upload finished. err: <nil> Duration: 30.003272211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.XX] duration: 30.002256887s trace: https://jaeger.dev.storj.io/trace/19c197d2080969ab FAIL (2025-12-28 01:34:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 44.336697838s) [upload=471896]: Uploading segments... Dec 28 01:33:44.465724151 (538h4m44.336807226s) [upload=471896]: Got next segment. Inline: false Dec 28 01:33:44.465776710 (538h4m44.336859746s) [upload=471896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:33:44.465782011 (538h4m44.336865046s) [upload=471896]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:33:44.715057114 (538h4m44.586140159s) [seg_pos={0 0}, upload=471896]: Begin upload segment... Dec 28 01:33:44.715121587 (538h4m44.586204632s) [seg_pos={0 0}, upload=471896]: Done begin upload segment. Dec 28 01:33:44.715149911 (538h4m44.586232956s) [noise=true , address=145.79.227.4:4186 , piece_id=EAKCIYXSLVSTY7OIZBJFCF2CIJY67COFHILILLHCMZUUNKRHNGFA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.715194426 (538h4m44.586277471s) [noise=true , address=77.90.31.10:4517 , piece_id=PWORLPA5IB4KVB5IFLFNAPWMANJ66PRDMWXEHNK2OBYJJHR64JEA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.715813355 (538h4m44.58689653s) [noise=true , address=145.79.227.39:4015 , piece_id=KIH4EGKABDSRBHEQR5JNSWLUBZSRGHADFULAD3SUHGMQYMKE5AAA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.716234015 (538h4m44.587317061s) [noise=true , address=77.90.31.14:4663 , piece_id=QS2HLJ5SHXX5MTQ6UMKOTEE4JZSNQ6DDV5FFGC7MCO7MLC3SAA4Q, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.716655328 (538h4m44.587738523s) [noise=true , address=77.90.31.15:4697 , piece_id=MHTPVX7QTU6BZXCXWFIAYYBSDF36MSGCY7CJXGK4RPNWFDJEFW4Q, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.717119362 (538h4m44.588202397s) [noise=true , address=77.90.31.18:4808 , piece_id=SI3BUZXBRIGUD6TR7MNPRTJGHLDZ76DJWCEE3HFNZQRAMC2A3NIQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.717584497 (538h4m44.588667542s) [noise=true , address=145.79.227.41:4087 , piece_id=7WLGORYTBEXUW55HTFV7L272EFE6OXN2MIH7ND4A4KFXBGRQ5UKQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.719038847 (538h4m44.590121882s) [noise=true , address=145.79.227.52:4445 , piece_id=LO77CPZLZ2W6V6KFTIHVYU22ATYGLWOHMXE2D2ZWZHSBESMSMROQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.719468616 (538h4m44.590551661s) [noise=true , address=45.91.116.11:4119 , piece_id=3JS6D35SNXHFLSRPKYHXP2KVGERZXRNPRKTBIUPDKEY3H4LAVPRA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.720275562 (538h4m44.591358597s) [noise=true , address=145.79.227.14:4838 , piece_id=S6XPZZUDJKX5BXYNUP7M36A4EKMXOXYYWV5XPTZMYA53QMEXOQTA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.721443396 (538h4m44.592526431s) [noise=true , address=145.79.227.45:4231 , piece_id=3XHZKB27W3GV5X4S74CNTVQXQB5CMAUU4BM4JR57YQ6HYFDZA6DQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.724290718 (538h4m44.595373753s) [noise=true , address=145.79.227.31:4798 , piece_id=L47DFGKWLIXI7SIBBW3SIEBNVICWYOV7X4KRGHIUP7RANEVXYAIQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.726486601 (538h4m44.597569636s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5KST6PELZTWXWBEP34GPBFHVPBP7C6L5SAGVNBBJ6ZMMA5HBZAA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.730253583 (538h4m44.601336618s) [noise=true , address=45.91.116.10:4084 , piece_id=POWD72C7U4D33WPIXSL34MILKJCPQR5VCLYIIXPQRVXCMLZS4SWQ, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.730323096 (538h4m44.601406141s) [noise=true , address=145.79.227.18:4007 , piece_id=44WPGG3VVNMYRUXYMUUW3GSCUQTMR3GIWNCHLNTAB7FZFFVGXJCA, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:44.732054894 (538h4m44.603137939s) [noise=true , address=145.79.227.48:4303 , piece_id=JLIWTBJ7KJFXSLAGXT3R7IC5S3XGE3LDSH4ZW7BRGLCAFFXQNS4A, seg_pos={0 0}, upload=471896]: Uploading piece... Dec 28 01:33:46.434735452 (538h4m46.305818487s) [upload=471896]: Next returned nil segment Dec 28 01:33:46.434746554 (538h4m46.305829599s) [upload=471896]: Waiting for error group managing segments... Dec 28 01:33:46.726008074 (538h4m46.597091119s) [noise=true , address=145.79.227.48:4303 , piece_id=JLIWTBJ7KJFXSLAGXT3R7IC5S3XGE3LDSH4ZW7BRGLCAFFXQNS4A, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.727252113 (538h4m46.598335149s) [noise=true , address=145.79.227.31:4798 , piece_id=L47DFGKWLIXI7SIBBW3SIEBNVICWYOV7X4KRGHIUP7RANEVXYAIQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.730858691 (538h4m46.601941736s) [noise=true , address=145.79.227.39:4015 , piece_id=KIH4EGKABDSRBHEQR5JNSWLUBZSRGHADFULAD3SUHGMQYMKE5AAA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.748137636 (538h4m46.619220671s) [noise=true , address=145.79.227.45:4231 , piece_id=3XHZKB27W3GV5X4S74CNTVQXQB5CMAUU4BM4JR57YQ6HYFDZA6DQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.750381539 (538h4m46.621464585s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5KST6PELZTWXWBEP34GPBFHVPBP7C6L5SAGVNBBJ6ZMMA5HBZAA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.752552985 (538h4m46.62363602s) [noise=true , address=145.79.227.14:4838 , piece_id=S6XPZZUDJKX5BXYNUP7M36A4EKMXOXYYWV5XPTZMYA53QMEXOQTA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.762491962 (538h4m46.633575007s) [noise=true , address=145.79.227.52:4445 , piece_id=LO77CPZLZ2W6V6KFTIHVYU22ATYGLWOHMXE2D2ZWZHSBESMSMROQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.763241419 (538h4m46.634324454s) [noise=true , address=145.79.227.41:4087 , piece_id=7WLGORYTBEXUW55HTFV7L272EFE6OXN2MIH7ND4A4KFXBGRQ5UKQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:46.982947624 (538h4m46.85403067s) [noise=true , address=77.90.31.10:4517 , piece_id=PWORLPA5IB4KVB5IFLFNAPWMANJ66PRDMWXEHNK2OBYJJHR64JEA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:47.056108146 (538h4m46.927191191s) [noise=true , address=77.90.31.15:4697 , piece_id=MHTPVX7QTU6BZXCXWFIAYYBSDF36MSGCY7CJXGK4RPNWFDJEFW4Q, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:47.068689446 (538h4m46.939772491s) [noise=true , address=45.91.116.10:4084 , piece_id=POWD72C7U4D33WPIXSL34MILKJCPQR5VCLYIIXPQRVXCMLZS4SWQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:47.349881169 (538h4m47.220964214s) [noise=true , address=77.90.31.18:4808 , piece_id=SI3BUZXBRIGUD6TR7MNPRTJGHLDZ76DJWCEE3HFNZQRAMC2A3NIQ, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:47.839375743 (538h4m47.710458788s) [noise=true , address=45.91.116.11:4119 , piece_id=3JS6D35SNXHFLSRPKYHXP2KVGERZXRNPRKTBIUPDKEY3H4LAVPRA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: <nil> Dec 28 01:33:47.839400781 (538h4m47.710483826s) [seg_pos={0 0}, upload=471896]: Segment reached optimal threshold of 13 pieces. Dec 28 01:33:47.839624076 (538h4m47.710707111s) [noise=true , address=145.79.227.4:4186 , piece_id=EAKCIYXSLVSTY7OIZBJFCF2CIJY67COFHILILLHCMZUUNKRHNGFA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:59454->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:59454->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59454->145.79.227.4:4186: use of closed network connection Dec 28 01:33:47.839633184 (538h4m47.710716229s) [noise=true , address=145.79.227.4:4186 , piece_id=EAKCIYXSLVSTY7OIZBJFCF2CIJY67COFHILILLHCMZUUNKRHNGFA, seg_pos={0 0}, upload=471896]: Long tail cancellation due to optimal threshold Dec 28 01:33:47.839770285 (538h4m47.71085333s) [noise=true , address=145.79.227.18:4007 , piece_id=44WPGG3VVNMYRUXYMUUW3GSCUQTMR3GIWNCHLNTAB7FZFFVGXJCA, seg_pos={0 0}, upload=471896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:40114->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:40114->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40114->145.79.227.18:4007: use of closed network connection Dec 28 01:33:47.839779762 (538h4m47.710862798s) [noise=true , address=145.79.227.18:4007 , piece_id=44WPGG3VVNMYRUXYMUUW3GSCUQTMR3GIWNCHLNTAB7FZFFVGXJCA, seg_pos={0 0}, upload=471896]: Long tail cancellation due to optimal threshold Dec 28 01:33:47.839818105 (538h4m47.71090114s) [noise=true , address=77.90.31.14:4663 , piece_id=QS2HLJ5SHXX5MTQ6UMKOTEE4JZSNQ6DDV5FFGC7MCO7MLC3SAA4Q, seg_pos={0 0}, upload=471896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 01:33:47.839826782 (538h4m47.710909817s) [noise=true , address=77.90.31.14:4663 , piece_id=QS2HLJ5SHXX5MTQ6UMKOTEE4JZSNQ6DDV5FFGC7MCO7MLC3SAA4Q, seg_pos={0 0}, upload=471896]: Long tail cancellation due to optimal threshold Dec 28 01:33:47.839904150 (538h4m47.710987195s) [seg_pos={0 0}, upload=471896]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:33:47.840012897 (538h4m47.711095942s) [upload=471896]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:33:47.840018557 (538h4m47.711101602s) [upload=471896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:33:48.144187897 (538h4m48.015270932s) [upload=471896]: Done uploading segments... Dec 28 01:33:48.144206143 (538h4m48.015289178s) [upload=471896]: Upload finished. err: <nil> Duration: 30.002281513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.X] duration: 30.002604853s trace: https://jaeger.dev.storj.io/trace/6201288d61ef5614 FAIL (2025-12-28 01:33:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Dec 28 01:32:39.763203662 (538h3m39.634286707s) [upload=471881]: Starting upload Dec 28 01:32:39.763474148 (538h3m39.634557203s) [upload=471881]: Done starting upload Dec 28 01:32:39.763630957 (538h3m39.634713992s) [upload=471881]: Uploading segments... Dec 28 01:32:39.763778728 (538h3m39.634861763s) [upload=471881]: Got next segment. Inline: false Dec 28 01:32:39.763835837 (538h3m39.634918872s) [upload=471881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:32:39.763841588 (538h3m39.634924623s) [upload=471881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:32:40.009322988 (538h3m39.880406023s) [seg_pos={0 0}, upload=471881]: Begin upload segment... Dec 28 01:32:40.009375277 (538h3m39.880458312s) [seg_pos={0 0}, upload=471881]: Done begin upload segment. Dec 28 01:32:40.009441954 (538h3m39.880524999s) [noise=true , address=45.91.116.13:4154 , piece_id=WV4WEJ2AMETZ4NZFAIDIHI3AP2GUMHZWWWVWJK3YHBM6F3AR2EOQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.009654458 (538h3m39.880737503s) [noise=true , address=77.90.31.14:4663 , piece_id=EG3IKM3C6UMS727JRXCBDLIQCS4VU5ZBARRBXKVN7YHISW5LZLQA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.010475933 (538h3m39.881558968s) [noise=true , address=145.79.227.42:4123 , piece_id=2QO4V6YXQ436SNUIMP56XLCGR43WFG6J25O3WADEAHERZ4S4PJ6A, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.010723705 (538h3m39.88180675s) [noise=true , address=145.79.227.16:4909 , piece_id=2ROYIOZUDRWUTB4OV43OVBYC7ALOTQS4IB5EKBQXKAK7NGIECFSQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.011335600 (538h3m39.882418645s) [noise=true , address=145.79.227.43:4158 , piece_id=R7KEYA7L362EXECT5AKJQLLJRFOSWZ3BETGXS5MBBSMUDQ24E6MA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.011913189 (538h3m39.882996235s) [noise=true , address=145.79.227.52:4445 , piece_id=E7SZCOKTKAQK6II2JUXHAKSDNVFSSR4SLHQG4NDVA74OASIIBUCA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.012740285 (538h3m39.88382332s) [noise=true , address=77.90.31.11:4555 , piece_id=LDHBFDKVKBLBF7G3C4E54DL2SIW7MKFX4ZIYROGR3Y5YI3LJL3XQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.013142040 (538h3m39.884225085s) [noise=true , address=145.79.227.6:4259 , piece_id=TQKTXBVIXLTMW5HYPXHDFNR72UKWFHTKTS4PLOJ3N72HUAKXGHGQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.013895645 (538h3m39.88497869s) [noise=true , address=77.90.31.12:4590 , piece_id=NTI4YCV7MBXE7UYCB3OONW236QA7DIUZSZK6YZKO7VYHDVBEZVDA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.014391570 (538h3m39.885474605s) [noise=true , address=145.79.227.15:4876 , piece_id=BW2HPBEBX6NXUCFRKDKAT6P2KCQRI4KE2SN46WSQOZI5YR36L6AQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.014873828 (538h3m39.885956873s) [noise=true , address=145.79.227.30:4762 , piece_id=UREXYVFBEQPLGL2NGGPBLPXJTCI7TVD2CUUHQNPODVNIF5JQGDEQ, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.014910237 (538h3m39.885993282s) [noise=true , address=77.90.31.21:4878 , piece_id=U7C64MF35R2OB7UK7JUIDQ3YWNFLQJ5DF5ILXDYG4RCS6NPMEE5Q, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.015457229 (538h3m39.886540264s) [noise=true , address=145.79.227.14:4840 , piece_id=H6HQIYCQXSPLNT5ZTAK76EZMICL27I3DFNXJ4DJPUJHR4GHO535A, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.015872149 (538h3m39.886955184s) [noise=true , address=145.79.227.5:4222 , piece_id=3O2B2KIL5VOQA3IYCEL6DD64VBQNTBCCHNE3W2ELRWBTPOP7J74Q, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.016321896 (538h3m39.887404931s) [noise=true , address=145.79.227.2:4115 , piece_id=GTYG324DAPZO7XB27LOFEQUPVGRRE65MDRG4AVUW6RW7PSJKE7OA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:40.016734832 (538h3m39.887817877s) [noise=true , address=145.79.227.33:4871 , piece_id=GNAMGVDAIZH522BEFXFE746FPMYHGQ6VSDH2GDML4Y63YRXWLYAA, seg_pos={0 0}, upload=471881]: Uploading piece... Dec 28 01:32:41.744131848 (538h3m41.615214893s) [upload=471881]: Next returned nil segment Dec 28 01:32:41.744149712 (538h3m41.615232757s) [upload=471881]: Waiting for error group managing segments... Dec 28 01:32:42.020324773 (538h3m41.891407818s) [noise=true , address=145.79.227.52:4445 , piece_id=E7SZCOKTKAQK6II2JUXHAKSDNVFSSR4SLHQG4NDVA74OASIIBUCA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.034145222 (538h3m41.905228257s) [noise=true , address=145.79.227.16:4909 , piece_id=2ROYIOZUDRWUTB4OV43OVBYC7ALOTQS4IB5EKBQXKAK7NGIECFSQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.048348649 (538h3m41.919431694s) [noise=true , address=145.79.227.15:4876 , piece_id=BW2HPBEBX6NXUCFRKDKAT6P2KCQRI4KE2SN46WSQOZI5YR36L6AQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.052003890 (538h3m41.923086926s) [noise=true , address=145.79.227.42:4123 , piece_id=2QO4V6YXQ436SNUIMP56XLCGR43WFG6J25O3WADEAHERZ4S4PJ6A, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.084870290 (538h3m41.955953335s) [noise=true , address=145.79.227.14:4840 , piece_id=H6HQIYCQXSPLNT5ZTAK76EZMICL27I3DFNXJ4DJPUJHR4GHO535A, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.159573459 (538h3m42.030656504s) [noise=true , address=145.79.227.43:4158 , piece_id=R7KEYA7L362EXECT5AKJQLLJRFOSWZ3BETGXS5MBBSMUDQ24E6MA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.180047810 (538h3m42.051130855s) [noise=true , address=77.90.31.12:4590 , piece_id=NTI4YCV7MBXE7UYCB3OONW236QA7DIUZSZK6YZKO7VYHDVBEZVDA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.188561542 (538h3m42.059644587s) [noise=true , address=145.79.227.5:4222 , piece_id=3O2B2KIL5VOQA3IYCEL6DD64VBQNTBCCHNE3W2ELRWBTPOP7J74Q, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.240210880 (538h3m42.111293915s) [noise=true , address=145.79.227.2:4115 , piece_id=GTYG324DAPZO7XB27LOFEQUPVGRRE65MDRG4AVUW6RW7PSJKE7OA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:42.623484913 (538h3m42.494567948s) [noise=true , address=77.90.31.11:4555 , piece_id=LDHBFDKVKBLBF7G3C4E54DL2SIW7MKFX4ZIYROGR3Y5YI3LJL3XQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:43.270428102 (538h3m43.141511137s) [noise=true , address=77.90.31.21:4878 , piece_id=U7C64MF35R2OB7UK7JUIDQ3YWNFLQJ5DF5ILXDYG4RCS6NPMEE5Q, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:43.365901546 (538h3m43.236984591s) [noise=true , address=145.79.227.33:4871 , piece_id=GNAMGVDAIZH522BEFXFE746FPMYHGQ6VSDH2GDML4Y63YRXWLYAA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:43.376045032 (538h3m43.247128067s) [noise=true , address=145.79.227.30:4762 , piece_id=UREXYVFBEQPLGL2NGGPBLPXJTCI7TVD2CUUHQNPODVNIF5JQGDEQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: <nil> Dec 28 01:32:43.376064840 (538h3m43.247147875s) [seg_pos={0 0}, upload=471881]: Segment reached optimal threshold of 13 pieces. Dec 28 01:32:43.376256315 (538h3m43.24733935s) [noise=true , address=145.79.227.6:4259 , piece_id=TQKTXBVIXLTMW5HYPXHDFNR72UKWFHTKTS4PLOJ3N72HUAKXGHGQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:51502->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:51502->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51502->145.79.227.6:4259: use of closed network connection Dec 28 01:32:43.376264089 (538h3m43.247347134s) [noise=true , address=145.79.227.6:4259 , piece_id=TQKTXBVIXLTMW5HYPXHDFNR72UKWFHTKTS4PLOJ3N72HUAKXGHGQ, seg_pos={0 0}, upload=471881]: Long tail cancellation due to optimal threshold Dec 28 01:32:43.376408865 (538h3m43.24749191s) [noise=true , address=45.91.116.13:4154 , piece_id=WV4WEJ2AMETZ4NZFAIDIHI3AP2GUMHZWWWVWJK3YHBM6F3AR2EOQ, seg_pos={0 0}, upload=471881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 01:32:43.376415117 (538h3m43.247498152s) [noise=true , address=45.91.116.13:4154 , piece_id=WV4WEJ2AMETZ4NZFAIDIHI3AP2GUMHZWWWVWJK3YHBM6F3AR2EOQ, seg_pos={0 0}, upload=471881]: Long tail cancellation due to optimal threshold Dec 28 01:32:43.376453460 (538h3m43.247536505s) [noise=true , address=77.90.31.14:4663 , piece_id=EG3IKM3C6UMS727JRXCBDLIQCS4VU5ZBARRBXKVN7YHISW5LZLQA, seg_pos={0 0}, upload=471881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 01:32:43.376471414 (538h3m43.247554449s) [noise=true , address=77.90.31.14:4663 , piece_id=EG3IKM3C6UMS727JRXCBDLIQCS4VU5ZBARRBXKVN7YHISW5LZLQA, seg_pos={0 0}, upload=471881]: Long tail cancellation due to optimal threshold Dec 28 01:32:43.376541277 (538h3m43.247624312s) [seg_pos={0 0}, upload=471881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:32:43.376647760 (538h3m43.247730795s) [upload=471881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:32:43.376660905 (538h3m43.24774394s) [upload=471881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:32:43.665032081 (538h3m43.536115126s) [upload=471881]: Done uploading segments... Dec 28 01:32:43.665053582 (538h3m43.536136627s) [upload=471881]: Upload finished. err: <nil> Duration: 30.002627936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX.] duration: 6.586029633s trace: https://jaeger.dev.storj.io/trace/71a9b689d85528b1 FAIL (2025-12-28 01:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA) context deadline exceeded ======LOGS====== =471863]: Got next segment. Inline: false Dec 28 01:31:12.971720993 (538h2m12.842804038s) [upload=471863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:31:12.971727445 (538h2m12.84281049s) [upload=471863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:31:13.214164604 (538h2m13.08524764s) [seg_pos={0 0}, upload=471863]: Begin upload segment... Dec 28 01:31:13.214209571 (538h2m13.085292606s) [seg_pos={0 0}, upload=471863]: Done begin upload segment. Dec 28 01:31:13.214242113 (538h2m13.085325158s) [noise=true , address=145.79.227.11:4731 , piece_id=KDTCRPHO6525BDFMVJX2ACOECKEUJD5AQCLBSOPRHY6RDDRNS2SQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.214252382 (538h2m13.085335417s) [noise=true , address=45.91.116.8:4047 , piece_id=ILO5INCQMHP3U35M6XLGTHQVSNNHJ5KTEKXN6NFNOOI4ISI7R2VA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.214811517 (538h2m13.085894572s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.215149400 (538h2m13.086232435s) [noise=true , address=145.79.227.41:4086 , piece_id=KCPS6WJ2RIVNWSFSJKBGEVCFI3L5MMXCBBIYHZCLKZIKYCMSMKKQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.215546267 (538h2m13.086629302s) [noise=true , address=145.79.227.38:4977 , piece_id=OAWUXLPIKPXXOG33BOIGDNV4SMVW5KXMFJWGP2RWBD5CRKELKPCA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216021721 (538h2m13.087104766s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216382910 (538h2m13.087465955s) [noise=true , address=45.91.116.9:4264 , piece_id=QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216822556 (538h2m13.087905591s) [noise=true , address=145.79.227.12:4767 , piece_id=VJ7P45RHCO6VBNBWMDG2BXLL2MQRMLOPWOFRNZ7JT4A7E4UU7JJQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.217348208 (538h2m13.088431253s) [noise=true , address=45.91.116.6:4974 , piece_id=OXMGL7Q3RBLPOC74REMTLUIHM7SYJD4DGCZIJS4YAVKI2P6WKUZA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.218172518 (538h2m13.089255553s) [noise=true , address=45.91.116.7:4009 , piece_id=Y5B4GCKI2EY6VV5TAS5X56JBBQQOBWEND5LM7BTU6SGHFLLJUAKQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.219705187 (538h2m13.090788232s) [noise=true , address=77.90.31.19:4844 , piece_id=N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.221575027 (538h2m13.092658062s) [noise=true , address=145.79.227.46:4602 , piece_id=ZCBQRASST2GQR32O764BV5R6VKFEXOJASWDGSVQOAYM2ZL6SIOYA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.222795752 (538h2m13.093878787s) [noise=true , address=145.79.227.15:4876 , piece_id=5QEWJK2WP5CGGX3O5PQTJL63KGQYOTSZIXBM2WKHY22X6IBIAH5A, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.224352947 (538h2m13.095435992s) [noise=true , address=77.90.31.14:4663 , piece_id=DIFOHEZPO22ZWUQ5GHAQBCFR6DRJZOKHCLOZAIUE7HDISOZVZ7IQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.225722184 (538h2m13.096805219s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.227124285 (538h2m13.09820732s) [noise=true , address=77.90.31.11:4554 , piece_id=ALSETAC3LVGWVK2VNOYMIYNZPKDUVXEETZ45V6FDZLA4QIVR2WVA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:14.942301011 (538h2m14.813384056s) [upload=471863]: Next returned nil segment Dec 28 01:31:14.942329616 (538h2m14.813412651s) [upload=471863]: Waiting for error group managing segments... Dec 28 01:31:15.223668791 (538h2m15.094751836s) [noise=true , address=145.79.227.11:4731 , piece_id=KDTCRPHO6525BDFMVJX2ACOECKEUJD5AQCLBSOPRHY6RDDRNS2SQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.225117469 (538h2m15.096200504s) [noise=true , address=145.79.227.41:4086 , piece_id=KCPS6WJ2RIVNWSFSJKBGEVCFI3L5MMXCBBIYHZCLKZIKYCMSMKKQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.227470730 (538h2m15.098553796s) [noise=true , address=145.79.227.38:4977 , piece_id=OAWUXLPIKPXXOG33BOIGDNV4SMVW5KXMFJWGP2RWBD5CRKELKPCA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.230745497 (538h2m15.101828542s) [noise=true , address=145.79.227.15:4876 , piece_id=5QEWJK2WP5CGGX3O5PQTJL63KGQYOTSZIXBM2WKHY22X6IBIAH5A, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.245920256 (538h2m15.117003301s) [noise=true , address=145.79.227.46:4602 , piece_id=ZCBQRASST2GQR32O764BV5R6VKFEXOJASWDGSVQOAYM2ZL6SIOYA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.256571719 (538h2m15.127654754s) [noise=true , address=145.79.227.12:4767 , piece_id=VJ7P45RHCO6VBNBWMDG2BXLL2MQRMLOPWOFRNZ7JT4A7E4UU7JJQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.466823640 (538h2m15.337906685s) [noise=true , address=45.91.116.7:4009 , piece_id=Y5B4GCKI2EY6VV5TAS5X56JBBQQOBWEND5LM7BTU6SGHFLLJUAKQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.499117488 (538h2m15.370200533s) [noise=true , address=45.91.116.9:4264 , piece_id=QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.511416660 (538h2m15.382499695s) [noise=true , address=77.90.31.11:4554 , piece_id=ALSETAC3LVGWVK2VNOYMIYNZPKDUVXEETZ45V6FDZLA4QIVR2WVA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.556860059 (538h2m15.427943094s) [noise=true , address=45.91.116.6:4974 , piece_id=OXMGL7Q3RBLPOC74REMTLUIHM7SYJD4DGCZIJS4YAVKI2P6WKUZA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.571902576 (538h2m15.442985621s) [noise=true , address=77.90.31.14:4663 , piece_id=DIFOHEZPO22ZWUQ5GHAQBCFR6DRJZOKHCLOZAIUE7HDISOZVZ7IQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.863484746 (538h2m15.734567781s) [noise=true , address=77.90.31.19:4844 , piece_id=N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:16.164587227 (538h2m16.035670263s) [noise=true , address=45.91.116.8:4047 , piece_id=ILO5INCQMHP3U35M6XLGTHQVSNNHJ5KTEKXN6NFNOOI4ISI7R2VA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:16.164606193 (538h2m16.035689238s) [seg_pos={0 0}, upload=471863]: Segment reached optimal threshold of 13 pieces. Dec 28 01:31:16.164845900 (538h2m16.035928935s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection Dec 28 01:31:16.164854606 (538h2m16.035937641s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165162663 (538h2m16.036245698s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection Dec 28 01:31:16.165170007 (538h2m16.036253042s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165183372 (538h2m16.036266418s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:39486->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 01:31:16.165198361 (538h2m16.036281396s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165236533 (538h2m16.036319568s) [seg_pos={0 0}, upload=471863]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:31:16.165341553 (538h2m16.036424588s) [upload=471863]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:31:16.165346944 (538h2m16.036429979s) [upload=471863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:31:16.464029957 (538h2m16.335112992s) [upload=471863]: Done uploading segments... Dec 28 01:31:16.464052058 (538h2m16.335135093s) [upload=471863]: Upload finished. err: <nil> Duration: 6.58607463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.XX] duration: 30.002837555s trace: https://jaeger.dev.storj.io/trace/71a9b689d85528b1 FAIL (2025-12-28 01:31:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA) context deadline exceeded ======LOGS====== =471863]: Got next segment. Inline: false Dec 28 01:31:12.971720993 (538h2m12.842804038s) [upload=471863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:31:12.971727445 (538h2m12.84281049s) [upload=471863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:31:13.214164604 (538h2m13.08524764s) [seg_pos={0 0}, upload=471863]: Begin upload segment... Dec 28 01:31:13.214209571 (538h2m13.085292606s) [seg_pos={0 0}, upload=471863]: Done begin upload segment. Dec 28 01:31:13.214242113 (538h2m13.085325158s) [noise=true , address=145.79.227.11:4731 , piece_id=KDTCRPHO6525BDFMVJX2ACOECKEUJD5AQCLBSOPRHY6RDDRNS2SQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.214252382 (538h2m13.085335417s) [noise=true , address=45.91.116.8:4047 , piece_id=ILO5INCQMHP3U35M6XLGTHQVSNNHJ5KTEKXN6NFNOOI4ISI7R2VA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.214811517 (538h2m13.085894572s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.215149400 (538h2m13.086232435s) [noise=true , address=145.79.227.41:4086 , piece_id=KCPS6WJ2RIVNWSFSJKBGEVCFI3L5MMXCBBIYHZCLKZIKYCMSMKKQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.215546267 (538h2m13.086629302s) [noise=true , address=145.79.227.38:4977 , piece_id=OAWUXLPIKPXXOG33BOIGDNV4SMVW5KXMFJWGP2RWBD5CRKELKPCA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216021721 (538h2m13.087104766s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216382910 (538h2m13.087465955s) [noise=true , address=45.91.116.9:4264 , piece_id=QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.216822556 (538h2m13.087905591s) [noise=true , address=145.79.227.12:4767 , piece_id=VJ7P45RHCO6VBNBWMDG2BXLL2MQRMLOPWOFRNZ7JT4A7E4UU7JJQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.217348208 (538h2m13.088431253s) [noise=true , address=45.91.116.6:4974 , piece_id=OXMGL7Q3RBLPOC74REMTLUIHM7SYJD4DGCZIJS4YAVKI2P6WKUZA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.218172518 (538h2m13.089255553s) [noise=true , address=45.91.116.7:4009 , piece_id=Y5B4GCKI2EY6VV5TAS5X56JBBQQOBWEND5LM7BTU6SGHFLLJUAKQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.219705187 (538h2m13.090788232s) [noise=true , address=77.90.31.19:4844 , piece_id=N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.221575027 (538h2m13.092658062s) [noise=true , address=145.79.227.46:4602 , piece_id=ZCBQRASST2GQR32O764BV5R6VKFEXOJASWDGSVQOAYM2ZL6SIOYA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.222795752 (538h2m13.093878787s) [noise=true , address=145.79.227.15:4876 , piece_id=5QEWJK2WP5CGGX3O5PQTJL63KGQYOTSZIXBM2WKHY22X6IBIAH5A, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.224352947 (538h2m13.095435992s) [noise=true , address=77.90.31.14:4663 , piece_id=DIFOHEZPO22ZWUQ5GHAQBCFR6DRJZOKHCLOZAIUE7HDISOZVZ7IQ, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.225722184 (538h2m13.096805219s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:13.227124285 (538h2m13.09820732s) [noise=true , address=77.90.31.11:4554 , piece_id=ALSETAC3LVGWVK2VNOYMIYNZPKDUVXEETZ45V6FDZLA4QIVR2WVA, seg_pos={0 0}, upload=471863]: Uploading piece... Dec 28 01:31:14.942301011 (538h2m14.813384056s) [upload=471863]: Next returned nil segment Dec 28 01:31:14.942329616 (538h2m14.813412651s) [upload=471863]: Waiting for error group managing segments... Dec 28 01:31:15.223668791 (538h2m15.094751836s) [noise=true , address=145.79.227.11:4731 , piece_id=KDTCRPHO6525BDFMVJX2ACOECKEUJD5AQCLBSOPRHY6RDDRNS2SQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.225117469 (538h2m15.096200504s) [noise=true , address=145.79.227.41:4086 , piece_id=KCPS6WJ2RIVNWSFSJKBGEVCFI3L5MMXCBBIYHZCLKZIKYCMSMKKQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.227470730 (538h2m15.098553796s) [noise=true , address=145.79.227.38:4977 , piece_id=OAWUXLPIKPXXOG33BOIGDNV4SMVW5KXMFJWGP2RWBD5CRKELKPCA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.230745497 (538h2m15.101828542s) [noise=true , address=145.79.227.15:4876 , piece_id=5QEWJK2WP5CGGX3O5PQTJL63KGQYOTSZIXBM2WKHY22X6IBIAH5A, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.245920256 (538h2m15.117003301s) [noise=true , address=145.79.227.46:4602 , piece_id=ZCBQRASST2GQR32O764BV5R6VKFEXOJASWDGSVQOAYM2ZL6SIOYA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.256571719 (538h2m15.127654754s) [noise=true , address=145.79.227.12:4767 , piece_id=VJ7P45RHCO6VBNBWMDG2BXLL2MQRMLOPWOFRNZ7JT4A7E4UU7JJQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.466823640 (538h2m15.337906685s) [noise=true , address=45.91.116.7:4009 , piece_id=Y5B4GCKI2EY6VV5TAS5X56JBBQQOBWEND5LM7BTU6SGHFLLJUAKQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.499117488 (538h2m15.370200533s) [noise=true , address=45.91.116.9:4264 , piece_id=QKDFRSVH3434PX57CDOYP3RJTMN2BE2Z7YGRQRU7RXCA723JPEMA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.511416660 (538h2m15.382499695s) [noise=true , address=77.90.31.11:4554 , piece_id=ALSETAC3LVGWVK2VNOYMIYNZPKDUVXEETZ45V6FDZLA4QIVR2WVA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.556860059 (538h2m15.427943094s) [noise=true , address=45.91.116.6:4974 , piece_id=OXMGL7Q3RBLPOC74REMTLUIHM7SYJD4DGCZIJS4YAVKI2P6WKUZA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.571902576 (538h2m15.442985621s) [noise=true , address=77.90.31.14:4663 , piece_id=DIFOHEZPO22ZWUQ5GHAQBCFR6DRJZOKHCLOZAIUE7HDISOZVZ7IQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:15.863484746 (538h2m15.734567781s) [noise=true , address=77.90.31.19:4844 , piece_id=N2236YLXGO232RWDUCQ7TAIZMDXXY7RMWZB5X3OWQFBDCJNWZ3IQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:16.164587227 (538h2m16.035670263s) [noise=true , address=45.91.116.8:4047 , piece_id=ILO5INCQMHP3U35M6XLGTHQVSNNHJ5KTEKXN6NFNOOI4ISI7R2VA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: <nil> Dec 28 01:31:16.164606193 (538h2m16.035689238s) [seg_pos={0 0}, upload=471863]: Segment reached optimal threshold of 13 pieces. Dec 28 01:31:16.164845900 (538h2m16.035928935s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59052->145.79.227.18:4007: use of closed network connection Dec 28 01:31:16.164854606 (538h2m16.035937641s) [noise=true , address=145.79.227.18:4007 , piece_id=DD64YZJDTSJ5DKRCKEM6WHJD6VDFDDBYLHD4C5QQSETGB4IGY2UA, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165162663 (538h2m16.036245698s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.6:4257: use of closed network connection Dec 28 01:31:16.165170007 (538h2m16.036253042s) [noise=true , address=145.79.227.6:4257 , piece_id=ARMSG2RQZ475XDX6DDZTDZ4AZZ3JPKWJOWI7Q2FHYQNUOPGNKWEA, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165183372 (538h2m16.036266418s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:39486->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 01:31:16.165198361 (538h2m16.036281396s) [noise=true , address=45.91.116.12:4226 , piece_id=R6FQWV7E3ABVZ32SZE7HORXTHFDPIYRCAJRHEBDAN53QW3ASQAJQ, seg_pos={0 0}, upload=471863]: Long tail cancellation due to optimal threshold Dec 28 01:31:16.165236533 (538h2m16.036319568s) [seg_pos={0 0}, upload=471863]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:31:16.165341553 (538h2m16.036424588s) [upload=471863]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:31:16.165346944 (538h2m16.036429979s) [upload=471863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:31:16.464029957 (538h2m16.335112992s) [upload=471863]: Done uploading segments... Dec 28 01:31:16.464052058 (538h2m16.335135093s) [upload=471863]: Upload finished. err: <nil> Duration: 30.002859738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.X] duration: 30.002120407s trace: https://jaeger.dev.storj.io/trace/799500dbac547643 FAIL (2025-12-28 01:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:30:08.965776246 (538h1m8.836859311s) [upload=471848]: Uploading segments... Dec 28 01:30:08.965998889 (538h1m8.837081935s) [upload=471848]: Got next segment. Inline: false Dec 28 01:30:08.966084523 (538h1m8.837167568s) [upload=471848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:30:08.966091286 (538h1m8.837174331s) [upload=471848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:30:09.214429055 (538h1m9.0855121s) [seg_pos={0 0}, upload=471848]: Begin upload segment... Dec 28 01:30:09.214483368 (538h1m9.085566413s) [seg_pos={0 0}, upload=471848]: Done begin upload segment. Dec 28 01:30:09.214592106 (538h1m9.085675151s) [noise=true , address=145.79.227.15:4876 , piece_id=7WWNCLPAJ4TS24LUVSMDPRBG4RYQ5NU5F45OH3MO73ENE7Y4YXOA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.214836010 (538h1m9.085919065s) [noise=true , address=145.79.227.37:4567 , piece_id=OU3VSSC6QKKGKTOHAMAXCW3IKD63PPKWVOSF2CAQL4I2VFPJ2IMQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.215848889 (538h1m9.086931924s) [noise=true , address=45.91.116.7:4012 , piece_id=WSEOCOU6KW7EHWEXFQKJWCBBD6OEDL36QAIWUOO4UCD6DLM2VEEA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.216554402 (538h1m9.087637447s) [noise=true , address=145.79.227.14:4840 , piece_id=JRPN7L6QUOS5EGSO67I76ZZZAEGAOMH2RMOQDI7ICAECVFIA53AA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.217219809 (538h1m9.088302844s) [noise=true , address=45.91.116.6:4975 , piece_id=L5SHFRYNBX3F7JUTNDWBVVQZNAFJNTIWVT4GNBBWAEPYMGQGF4QA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.217378672 (538h1m9.088461707s) [noise=true , address=77.90.31.12:4589 , piece_id=7OLPXHTWAILPEM7QFJ4GI3ADZ7PWAHUGUXOG3JGQCZYSKJ3F7PRQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.217987972 (538h1m9.089071007s) [noise=true , address=45.91.116.11:4118 , piece_id=4NKPE55YLSPNU6EHGO2ZF56CCRHQXPASOKYLPAQMAFQ5ZEO6M3FA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.219677891 (538h1m9.090760946s) [noise=true , address=45.91.116.10:4081 , piece_id=CIAAWXE2E7BOROYIBT47DDPQVSVPHI6IN3KLV7YO2X7EA3VEV5XQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.220263475 (538h1m9.09134652s) [noise=true , address=145.79.227.52:4446 , piece_id=3ZOQN6NRFMOPECAVCGXS6DXGEWTCORX3GDUBYAZ652CSY5O6RCDQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.220981242 (538h1m9.092064277s) [noise=true , address=45.91.116.4:4939 , piece_id=5RPTI3EHA6REL3BUUDJSFRUUXE6XSII6XQJ6ZCBF7STES5RNIOEA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.221632142 (538h1m9.092715187s) [noise=true , address=45.91.116.13:4155 , piece_id=BK23KAQD3RWC2VQONVYNXWX7MMGQSTZB3FCVYAHSII2OZ4CVAXGQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.222299853 (538h1m9.093382898s) [noise=true , address=145.79.227.36:4519 , piece_id=FT34GYBSJBHIT56UBJWDEHU2V57KMFULNEP7TQL6FT3QD5RGOVMQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.223267876 (538h1m9.094350922s) [noise=true , address=45.91.116.5:4299 , piece_id=DC32N4LOGVMV7QQKQ24FLC7Q3K2GBC6WGMIJIPBMOA55BYWZSGQQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.224538887 (538h1m9.095621932s) [noise=true , address=145.79.227.45:4230 , piece_id=TXHQTIJM2OBUCCKDB7K5EIQZE2DEVSZ3F3QJXVYK27RFEOOKNTNA, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.225318972 (538h1m9.096402007s) [noise=true , address=45.91.116.12:4228 , piece_id=T3NRRSICYQFROG6DBHRFRK274IUJUVKVH22IWDFTHV4YOQHKO7OQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:09.225851938 (538h1m9.096934993s) [noise=true , address=77.90.31.17:4770 , piece_id=6SIOIXSWHDRVEGQ7RSAZAOJDPRJKOYDCVUHBCV2UKN3HDCEF2BZQ, seg_pos={0 0}, upload=471848]: Uploading piece... Dec 28 01:30:10.948174965 (538h1m10.81925801s) [upload=471848]: Next returned nil segment Dec 28 01:30:10.948212306 (538h1m10.819295351s) [upload=471848]: Waiting for error group managing segments... Dec 28 01:30:11.235741628 (538h1m11.106824663s) [noise=true , address=145.79.227.36:4519 , piece_id=FT34GYBSJBHIT56UBJWDEHU2V57KMFULNEP7TQL6FT3QD5RGOVMQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.250034186 (538h1m11.121117221s) [noise=true , address=145.79.227.37:4567 , piece_id=OU3VSSC6QKKGKTOHAMAXCW3IKD63PPKWVOSF2CAQL4I2VFPJ2IMQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.250378322 (538h1m11.121461367s) [noise=true , address=145.79.227.14:4840 , piece_id=JRPN7L6QUOS5EGSO67I76ZZZAEGAOMH2RMOQDI7ICAECVFIA53AA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.256616843 (538h1m11.127699888s) [noise=true , address=145.79.227.45:4230 , piece_id=TXHQTIJM2OBUCCKDB7K5EIQZE2DEVSZ3F3QJXVYK27RFEOOKNTNA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.262856014 (538h1m11.133939069s) [noise=true , address=145.79.227.52:4446 , piece_id=3ZOQN6NRFMOPECAVCGXS6DXGEWTCORX3GDUBYAZ652CSY5O6RCDQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.281500219 (538h1m11.152583274s) [noise=true , address=145.79.227.15:4876 , piece_id=7WWNCLPAJ4TS24LUVSMDPRBG4RYQ5NU5F45OH3MO73ENE7Y4YXOA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.456099935 (538h1m11.327182981s) [noise=true , address=45.91.116.10:4081 , piece_id=CIAAWXE2E7BOROYIBT47DDPQVSVPHI6IN3KLV7YO2X7EA3VEV5XQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.511986342 (538h1m11.383069387s) [noise=true , address=77.90.31.17:4770 , piece_id=6SIOIXSWHDRVEGQ7RSAZAOJDPRJKOYDCVUHBCV2UKN3HDCEF2BZQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.542950930 (538h1m11.414033975s) [noise=true , address=45.91.116.7:4012 , piece_id=WSEOCOU6KW7EHWEXFQKJWCBBD6OEDL36QAIWUOO4UCD6DLM2VEEA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.571437357 (538h1m11.442520423s) [noise=true , address=45.91.116.11:4118 , piece_id=4NKPE55YLSPNU6EHGO2ZF56CCRHQXPASOKYLPAQMAFQ5ZEO6M3FA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.571557006 (538h1m11.442640041s) [noise=true , address=45.91.116.5:4299 , piece_id=DC32N4LOGVMV7QQKQ24FLC7Q3K2GBC6WGMIJIPBMOA55BYWZSGQQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.649293531 (538h1m11.520376577s) [noise=true , address=77.90.31.12:4589 , piece_id=7OLPXHTWAILPEM7QFJ4GI3ADZ7PWAHUGUXOG3JGQCZYSKJ3F7PRQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.700036072 (538h1m11.571119117s) [noise=true , address=45.91.116.6:4975 , piece_id=L5SHFRYNBX3F7JUTNDWBVVQZNAFJNTIWVT4GNBBWAEPYMGQGF4QA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: <nil> Dec 28 01:30:11.700304945 (538h1m11.57138799s) [seg_pos={0 0}, upload=471848]: Segment reached optimal threshold of 13 pieces. Dec 28 01:30:11.700857226 (538h1m11.571940271s) [noise=true , address=45.91.116.12:4228 , piece_id=T3NRRSICYQFROG6DBHRFRK274IUJUVKVH22IWDFTHV4YOQHKO7OQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 01:30:11.700878167 (538h1m11.571961212s) [noise=true , address=45.91.116.12:4228 , piece_id=T3NRRSICYQFROG6DBHRFRK274IUJUVKVH22IWDFTHV4YOQHKO7OQ, seg_pos={0 0}, upload=471848]: Long tail cancellation due to optimal threshold Dec 28 01:30:11.702008689 (538h1m11.573091734s) [noise=true , address=45.91.116.4:4939 , piece_id=5RPTI3EHA6REL3BUUDJSFRUUXE6XSII6XQJ6ZCBF7STES5RNIOEA, seg_pos={0 0}, upload=471848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47970->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47970->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47970->45.91.116.4:4939: use of closed network connection Dec 28 01:30:11.702020682 (538h1m11.573103727s) [noise=true , address=45.91.116.4:4939 , piece_id=5RPTI3EHA6REL3BUUDJSFRUUXE6XSII6XQJ6ZCBF7STES5RNIOEA, seg_pos={0 0}, upload=471848]: Long tail cancellation due to optimal threshold Dec 28 01:30:11.702068623 (538h1m11.573151658s) [noise=true , address=45.91.116.13:4155 , piece_id=BK23KAQD3RWC2VQONVYNXWX7MMGQSTZB3FCVYAHSII2OZ4CVAXGQ, seg_pos={0 0}, upload=471848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:36684->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:36684->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->45.91.116.13:4155: use of closed network connection Dec 28 01:30:11.702076438 (538h1m11.573159483s) [noise=true , address=45.91.116.13:4155 , piece_id=BK23KAQD3RWC2VQONVYNXWX7MMGQSTZB3FCVYAHSII2OZ4CVAXGQ, seg_pos={0 0}, upload=471848]: Long tail cancellation due to optimal threshold Dec 28 01:30:11.702158315 (538h1m11.57324136s) [seg_pos={0 0}, upload=471848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:30:11.702615645 (538h1m11.57369868s) [upload=471848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:30:11.702624692 (538h1m11.573707727s) [upload=471848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:30:12.002901241 (538h1m11.873984316s) [upload=471848]: Done uploading segments... Dec 28 01:30:12.002927401 (538h1m11.874010446s) [upload=471848]: Upload finished. err: <nil> Duration: 30.002143671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X.] duration: 10.383242936s trace: https://jaeger.dev.storj.io/trace/3058b7cbc57c2ee6 FAIL (2025-12-28 01:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 28 01:28:52.598266241 (537h59m52.469349276s) [upload=471830]: Got next segment. Inline: false Dec 28 01:28:52.598310435 (537h59m52.469393481s) [upload=471830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:28:52.598316026 (537h59m52.469399071s) [upload=471830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:28:52.840111728 (537h59m52.711194763s) [seg_pos={0 0}, upload=471830]: Begin upload segment... Dec 28 01:28:52.840169417 (537h59m52.711252462s) [seg_pos={0 0}, upload=471830]: Done begin upload segment. Dec 28 01:28:52.840216828 (537h59m52.711299873s) [noise=true , address=45.91.116.5:4297 , piece_id=3QWIVESAMXBQKB4PKMXO3KOIKEOI3DKCG2GQWKK6IDRNCW5J4GEA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.840284907 (537h59m52.711367952s) [noise=true , address=45.91.116.6:4975 , piece_id=CWOMMTLUW7OF65YWOA5AHV2DN3UVE2SBHTYEVLU3AHOHYUGW4U7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.840925898 (537h59m52.712008943s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841021951 (537h59m52.712104986s) [noise=true , address=45.91.116.9:4262 , piece_id=FKSWHE4A3JCWTW6IQR65JLB6MZMN3V4ILMYOAZQHNWUGYPFFEY7A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841377958 (537h59m52.712460993s) [noise=true , address=145.79.227.15:4876 , piece_id=57F3YFPHZ5DKTJAEKXPTYOZZDNLIDZJDKXRF4HB5MIGPQNZYTKMQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841577639 (537h59m52.712660684s) [noise=true , address=145.79.227.46:4603 , piece_id=IEIIM6XS6VBVTCIT7YY4TN2KQUVGNTQXIHJRGCFOW2EFJE6YUACQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841667210 (537h59m52.712750245s) [noise=true , address=77.90.31.10:4519 , piece_id=XYTPRBQY56RAHYKQVFE53RKPQDBT2MRVNCCW6YIDS6QQZQJJ3QTA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842020723 (537h59m52.713103768s) [noise=true , address=45.91.116.7:4011 , piece_id=MQ2NGCMSD4T7TSCQZSWGFXKVYE7T2ET2CP6P5GKRU5OKIMRH2SYQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842209923 (537h59m52.713292978s) [noise=true , address=145.79.227.48:4303 , piece_id=ZKN6NE5ACR6SJFMGELDLSOMYRF6KYVYKJPK5UAZL6FNNMXHWLRRQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842396408 (537h59m52.713479453s) [noise=true , address=145.79.227.47:4267 , piece_id=UWDLKNKFIZSJF3PB5PYD7DSLXTE52RIXEP54QXFFW7BXYCOUOHDA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842891671 (537h59m52.713974716s) [noise=true , address=145.79.227.38:4977 , piece_id=K2XLQDMT4H3UBUUYX4ZXC6IFWZJQYIOE7MNK45COJ3UHEXVDXUZQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842918322 (537h59m52.714001367s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.843553061 (537h59m52.714636106s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.844190905 (537h59m52.71527394s) [noise=true , address=145.79.227.31:4798 , piece_id=6PD6QUOAI3K3MUTFP4XWFYARDS7YPM5TZCM2F7WHOLQATZLEVR2A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.844869367 (537h59m52.715952412s) [noise=true , address=145.79.227.11:4730 , piece_id=AKR6ZDD52RXKKDLY4DMUDEOX2KWDOWD7BZP7KQSZDHOIOZ2H3E7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.845144952 (537h59m52.716227997s) [noise=true , address=145.79.227.51:4409 , piece_id=KWQKG5BIEXDF42CUP3SX7EPBMZGW57HJVQRLSHRW6ESOAAD43Z7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:54.548338889 (537h59m54.419421944s) [upload=471830]: Next returned nil segment Dec 28 01:28:54.548357383 (537h59m54.419440428s) [upload=471830]: Waiting for error group managing segments... Dec 28 01:28:54.834778306 (537h59m54.705861351s) [noise=true , address=145.79.227.11:4730 , piece_id=AKR6ZDD52RXKKDLY4DMUDEOX2KWDOWD7BZP7KQSZDHOIOZ2H3E7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.840873694 (537h59m54.711956739s) [noise=true , address=145.79.227.46:4603 , piece_id=IEIIM6XS6VBVTCIT7YY4TN2KQUVGNTQXIHJRGCFOW2EFJE6YUACQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.848003772 (537h59m54.719086807s) [noise=true , address=145.79.227.48:4303 , piece_id=ZKN6NE5ACR6SJFMGELDLSOMYRF6KYVYKJPK5UAZL6FNNMXHWLRRQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.849595774 (537h59m54.720678819s) [noise=true , address=145.79.227.31:4798 , piece_id=6PD6QUOAI3K3MUTFP4XWFYARDS7YPM5TZCM2F7WHOLQATZLEVR2A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.852974567 (537h59m54.724057602s) [noise=true , address=145.79.227.15:4876 , piece_id=57F3YFPHZ5DKTJAEKXPTYOZZDNLIDZJDKXRF4HB5MIGPQNZYTKMQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.856908169 (537h59m54.727991204s) [noise=true , address=145.79.227.47:4267 , piece_id=UWDLKNKFIZSJF3PB5PYD7DSLXTE52RIXEP54QXFFW7BXYCOUOHDA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.863421772 (537h59m54.734504808s) [noise=true , address=145.79.227.38:4977 , piece_id=K2XLQDMT4H3UBUUYX4ZXC6IFWZJQYIOE7MNK45COJ3UHEXVDXUZQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.878404204 (537h59m54.749487249s) [noise=true , address=145.79.227.51:4409 , piece_id=KWQKG5BIEXDF42CUP3SX7EPBMZGW57HJVQRLSHRW6ESOAAD43Z7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.104749202 (537h59m54.975832237s) [noise=true , address=45.91.116.5:4297 , piece_id=3QWIVESAMXBQKB4PKMXO3KOIKEOI3DKCG2GQWKK6IDRNCW5J4GEA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.122569197 (537h59m54.993652232s) [noise=true , address=45.91.116.7:4011 , piece_id=MQ2NGCMSD4T7TSCQZSWGFXKVYE7T2ET2CP6P5GKRU5OKIMRH2SYQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.132448871 (537h59m55.003531906s) [noise=true , address=45.91.116.9:4262 , piece_id=FKSWHE4A3JCWTW6IQR65JLB6MZMN3V4ILMYOAZQHNWUGYPFFEY7A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.178734554 (537h59m55.049817589s) [noise=true , address=45.91.116.6:4975 , piece_id=CWOMMTLUW7OF65YWOA5AHV2DN3UVE2SBHTYEVLU3AHOHYUGW4U7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.218076064 (537h59m55.089159099s) [noise=true , address=77.90.31.10:4519 , piece_id=XYTPRBQY56RAHYKQVFE53RKPQDBT2MRVNCCW6YIDS6QQZQJJ3QTA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.218097736 (537h59m55.089180771s) [seg_pos={0 0}, upload=471830]: Segment reached optimal threshold of 13 pieces. Dec 28 01:28:55.218301183 (537h59m55.089384229s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:28:55.218309519 (537h59m55.089392554s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218432342 (537h59m55.089515377s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection Dec 28 01:28:55.218438034 (537h59m55.089521069s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218870688 (537h59m55.089953733s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection Dec 28 01:28:55.218887119 (537h59m55.089970154s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218930852 (537h59m55.090013897s) [seg_pos={0 0}, upload=471830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:28:55.219149458 (537h59m55.090232503s) [upload=471830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:28:55.219158786 (537h59m55.090241831s) [upload=471830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:28:55.508452730 (537h59m55.379535775s) [upload=471830]: Done uploading segments... Dec 28 01:28:55.508471746 (537h59m55.379554781s) [upload=471830]: Upload finished. err: <nil> Duration: 10.383261672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XXXXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....X] duration: 30.003861408s trace: https://jaeger.dev.storj.io/trace/3058b7cbc57c2ee6 FAIL (2025-12-28 01:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 28 01:28:52.598266241 (537h59m52.469349276s) [upload=471830]: Got next segment. Inline: false Dec 28 01:28:52.598310435 (537h59m52.469393481s) [upload=471830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:28:52.598316026 (537h59m52.469399071s) [upload=471830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:28:52.840111728 (537h59m52.711194763s) [seg_pos={0 0}, upload=471830]: Begin upload segment... Dec 28 01:28:52.840169417 (537h59m52.711252462s) [seg_pos={0 0}, upload=471830]: Done begin upload segment. Dec 28 01:28:52.840216828 (537h59m52.711299873s) [noise=true , address=45.91.116.5:4297 , piece_id=3QWIVESAMXBQKB4PKMXO3KOIKEOI3DKCG2GQWKK6IDRNCW5J4GEA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.840284907 (537h59m52.711367952s) [noise=true , address=45.91.116.6:4975 , piece_id=CWOMMTLUW7OF65YWOA5AHV2DN3UVE2SBHTYEVLU3AHOHYUGW4U7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.840925898 (537h59m52.712008943s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841021951 (537h59m52.712104986s) [noise=true , address=45.91.116.9:4262 , piece_id=FKSWHE4A3JCWTW6IQR65JLB6MZMN3V4ILMYOAZQHNWUGYPFFEY7A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841377958 (537h59m52.712460993s) [noise=true , address=145.79.227.15:4876 , piece_id=57F3YFPHZ5DKTJAEKXPTYOZZDNLIDZJDKXRF4HB5MIGPQNZYTKMQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841577639 (537h59m52.712660684s) [noise=true , address=145.79.227.46:4603 , piece_id=IEIIM6XS6VBVTCIT7YY4TN2KQUVGNTQXIHJRGCFOW2EFJE6YUACQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.841667210 (537h59m52.712750245s) [noise=true , address=77.90.31.10:4519 , piece_id=XYTPRBQY56RAHYKQVFE53RKPQDBT2MRVNCCW6YIDS6QQZQJJ3QTA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842020723 (537h59m52.713103768s) [noise=true , address=45.91.116.7:4011 , piece_id=MQ2NGCMSD4T7TSCQZSWGFXKVYE7T2ET2CP6P5GKRU5OKIMRH2SYQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842209923 (537h59m52.713292978s) [noise=true , address=145.79.227.48:4303 , piece_id=ZKN6NE5ACR6SJFMGELDLSOMYRF6KYVYKJPK5UAZL6FNNMXHWLRRQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842396408 (537h59m52.713479453s) [noise=true , address=145.79.227.47:4267 , piece_id=UWDLKNKFIZSJF3PB5PYD7DSLXTE52RIXEP54QXFFW7BXYCOUOHDA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842891671 (537h59m52.713974716s) [noise=true , address=145.79.227.38:4977 , piece_id=K2XLQDMT4H3UBUUYX4ZXC6IFWZJQYIOE7MNK45COJ3UHEXVDXUZQ, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.842918322 (537h59m52.714001367s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.843553061 (537h59m52.714636106s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.844190905 (537h59m52.71527394s) [noise=true , address=145.79.227.31:4798 , piece_id=6PD6QUOAI3K3MUTFP4XWFYARDS7YPM5TZCM2F7WHOLQATZLEVR2A, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.844869367 (537h59m52.715952412s) [noise=true , address=145.79.227.11:4730 , piece_id=AKR6ZDD52RXKKDLY4DMUDEOX2KWDOWD7BZP7KQSZDHOIOZ2H3E7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:52.845144952 (537h59m52.716227997s) [noise=true , address=145.79.227.51:4409 , piece_id=KWQKG5BIEXDF42CUP3SX7EPBMZGW57HJVQRLSHRW6ESOAAD43Z7Q, seg_pos={0 0}, upload=471830]: Uploading piece... Dec 28 01:28:54.548338889 (537h59m54.419421944s) [upload=471830]: Next returned nil segment Dec 28 01:28:54.548357383 (537h59m54.419440428s) [upload=471830]: Waiting for error group managing segments... Dec 28 01:28:54.834778306 (537h59m54.705861351s) [noise=true , address=145.79.227.11:4730 , piece_id=AKR6ZDD52RXKKDLY4DMUDEOX2KWDOWD7BZP7KQSZDHOIOZ2H3E7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.840873694 (537h59m54.711956739s) [noise=true , address=145.79.227.46:4603 , piece_id=IEIIM6XS6VBVTCIT7YY4TN2KQUVGNTQXIHJRGCFOW2EFJE6YUACQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.848003772 (537h59m54.719086807s) [noise=true , address=145.79.227.48:4303 , piece_id=ZKN6NE5ACR6SJFMGELDLSOMYRF6KYVYKJPK5UAZL6FNNMXHWLRRQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.849595774 (537h59m54.720678819s) [noise=true , address=145.79.227.31:4798 , piece_id=6PD6QUOAI3K3MUTFP4XWFYARDS7YPM5TZCM2F7WHOLQATZLEVR2A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.852974567 (537h59m54.724057602s) [noise=true , address=145.79.227.15:4876 , piece_id=57F3YFPHZ5DKTJAEKXPTYOZZDNLIDZJDKXRF4HB5MIGPQNZYTKMQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.856908169 (537h59m54.727991204s) [noise=true , address=145.79.227.47:4267 , piece_id=UWDLKNKFIZSJF3PB5PYD7DSLXTE52RIXEP54QXFFW7BXYCOUOHDA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.863421772 (537h59m54.734504808s) [noise=true , address=145.79.227.38:4977 , piece_id=K2XLQDMT4H3UBUUYX4ZXC6IFWZJQYIOE7MNK45COJ3UHEXVDXUZQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:54.878404204 (537h59m54.749487249s) [noise=true , address=145.79.227.51:4409 , piece_id=KWQKG5BIEXDF42CUP3SX7EPBMZGW57HJVQRLSHRW6ESOAAD43Z7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.104749202 (537h59m54.975832237s) [noise=true , address=45.91.116.5:4297 , piece_id=3QWIVESAMXBQKB4PKMXO3KOIKEOI3DKCG2GQWKK6IDRNCW5J4GEA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.122569197 (537h59m54.993652232s) [noise=true , address=45.91.116.7:4011 , piece_id=MQ2NGCMSD4T7TSCQZSWGFXKVYE7T2ET2CP6P5GKRU5OKIMRH2SYQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.132448871 (537h59m55.003531906s) [noise=true , address=45.91.116.9:4262 , piece_id=FKSWHE4A3JCWTW6IQR65JLB6MZMN3V4ILMYOAZQHNWUGYPFFEY7A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.178734554 (537h59m55.049817589s) [noise=true , address=45.91.116.6:4975 , piece_id=CWOMMTLUW7OF65YWOA5AHV2DN3UVE2SBHTYEVLU3AHOHYUGW4U7Q, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.218076064 (537h59m55.089159099s) [noise=true , address=77.90.31.10:4519 , piece_id=XYTPRBQY56RAHYKQVFE53RKPQDBT2MRVNCCW6YIDS6QQZQJJ3QTA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: <nil> Dec 28 01:28:55.218097736 (537h59m55.089180771s) [seg_pos={0 0}, upload=471830]: Segment reached optimal threshold of 13 pieces. Dec 28 01:28:55.218301183 (537h59m55.089384229s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:28:55.218309519 (537h59m55.089392554s) [noise=true , address=45.91.116.4:4939 , piece_id=ORIYHWA2IJBHLAGSPDNN3WPRC3PIQUPSGKJUJPS4DGNBLVWQZU7A, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218432342 (537h59m55.089515377s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40040->77.90.31.13:4627: use of closed network connection Dec 28 01:28:55.218438034 (537h59m55.089521069s) [noise=true , address=77.90.31.13:4627 , piece_id=7KMSZ3ICK7PG35GKZIENECGBXCNDWNIWHRWSRIY73XJMIRFCRYFQ, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218870688 (537h59m55.089953733s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35650->45.91.116.10:4083: use of closed network connection Dec 28 01:28:55.218887119 (537h59m55.089970154s) [noise=true , address=45.91.116.10:4083 , piece_id=Z4CVZUYYEKKGCHGLDH54VIYULP6QKLRAZGXUJ5WE6KU7X74JDBKA, seg_pos={0 0}, upload=471830]: Long tail cancellation due to optimal threshold Dec 28 01:28:55.218930852 (537h59m55.090013897s) [seg_pos={0 0}, upload=471830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:28:55.219149458 (537h59m55.090232503s) [upload=471830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:28:55.219158786 (537h59m55.090241831s) [upload=471830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:28:55.508452730 (537h59m55.379535775s) [upload=471830]: Done uploading segments... Dec 28 01:28:55.508471746 (537h59m55.379554781s) [upload=471830]: Upload finished. err: <nil> Duration: 30.003873252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XXXXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX....] duration: 5.865959447s trace: https://jaeger.dev.storj.io/trace/f8c19cb1fcf31da FAIL (2025-12-28 01:28:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ) context deadline exceeded; piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ) context deadline exceeded ======LOGS====== 01:24:34.012568026 (537h55m33.883651071s) [upload=471768]: Starting upload Dec 28 01:24:34.012631917 (537h55m33.883714952s) [upload=471768]: Done starting upload Dec 28 01:24:34.012746135 (537h55m33.88382917s) [upload=471768]: Uploading segments... Dec 28 01:24:34.012831286 (537h55m33.883914321s) [upload=471768]: Got next segment. Inline: false Dec 28 01:24:34.012863478 (537h55m33.883946513s) [upload=471768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:24:34.012868617 (537h55m33.883951662s) [upload=471768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:24:34.256638478 (537h55m34.127721514s) [seg_pos={0 0}, upload=471768]: Begin upload segment... Dec 28 01:24:34.256680879 (537h55m34.127763914s) [seg_pos={0 0}, upload=471768]: Done begin upload segment. Dec 28 01:24:34.256770519 (537h55m34.127853554s) [noise=true , address=145.79.227.19:4079 , piece_id=CDVGGB5HESIDTTYAH5VN2DH3KWDTDZH233L4YEB4ED2M7O7DG7ZQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.256780649 (537h55m34.127863714s) [noise=true , address=145.79.227.14:4838 , piece_id=MECTRZB3J5WC2LQHANIA7STLXZGBS2F7OMRI6OKKZYXJ45TPXAEQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.258272680 (537h55m34.129355755s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.258887171 (537h55m34.129970206s) [noise=true , address=145.79.227.16:4909 , piece_id=4TSBXV4W2IFGBMZ7ILUE3XA3A5OHTDWB7KDIOXHR6MOGK5W2ZOCA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.259331486 (537h55m34.130414531s) [noise=true , address=77.90.31.10:4518 , piece_id=LLDCYP7P4HOMXJBBG3QBSJTGOQORKZ23JQX64WWGROM3PKJBAM5Q, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.260582909 (537h55m34.131665944s) [noise=true , address=77.90.31.13:4627 , piece_id=YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.260796106 (537h55m34.131879141s) [noise=true , address=45.91.116.5:4299 , piece_id=22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.261855403 (537h55m34.132938438s) [noise=true , address=145.79.227.51:4411 , piece_id=3SREH2KDJR23ZZJWUEPUZHU2ZLIQNN4QKGJUT3O4N75GKNZOAI2A, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.262546929 (537h55m34.133629975s) [noise=true , address=145.79.227.37:4565 , piece_id=TPY3OLVJ4OJ5N6WE4WUV25UEP5TYBGZQ4EBVXWPC76MUYDXODPBA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.263149177 (537h55m34.134232212s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.264078897 (537h55m34.135161942s) [noise=true , address=145.79.227.41:4087 , piece_id=V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.264619857 (537h55m34.135702892s) [noise=true , address=145.79.227.11:4731 , piece_id=XE2GQA37LFEYBRVMG5AMN275SS7XBQ6ZMF2M43R5MJXRHODEDD6Q, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.265493841 (537h55m34.136576876s) [noise=true , address=77.90.31.18:4808 , piece_id=Q2YDZRIECQXLO4QYZHPRNU35LGNO2R5R254X5MGACW5CIGXMVPWQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.266142537 (537h55m34.137225582s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.266841297 (537h55m34.137924342s) [noise=true , address=145.79.227.36:4518 , piece_id=3MGJJ3U5K4IED7BFX4SKJVRV4JB3H6ANCGDS63KICFH33H6F6YLQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.269263941 (537h55m34.140346976s) [noise=true , address=145.79.227.46:4601 , piece_id=GZJKDSQZ5DQYROTMBAVVRHJ7VGPWMJ6OTYNERXIAMODDMDFK77LA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:35.889960104 (537h55m35.761043139s) [upload=471768]: Next returned nil segment Dec 28 01:24:35.889971937 (537h55m35.761054982s) [upload=471768]: Waiting for error group managing segments... Dec 28 01:24:36.275593921 (537h55m36.146676957s) [noise=true , address=145.79.227.14:4838 , piece_id=MECTRZB3J5WC2LQHANIA7STLXZGBS2F7OMRI6OKKZYXJ45TPXAEQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.277215790 (537h55m36.148298825s) [noise=true , address=145.79.227.51:4411 , piece_id=3SREH2KDJR23ZZJWUEPUZHU2ZLIQNN4QKGJUT3O4N75GKNZOAI2A, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.279432271 (537h55m36.150515306s) [noise=true , address=145.79.227.11:4731 , piece_id=XE2GQA37LFEYBRVMG5AMN275SS7XBQ6ZMF2M43R5MJXRHODEDD6Q, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.282338966 (537h55m36.153422001s) [noise=true , address=145.79.227.41:4087 , piece_id=V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.282851623 (537h55m36.153934668s) [noise=true , address=145.79.227.37:4565 , piece_id=TPY3OLVJ4OJ5N6WE4WUV25UEP5TYBGZQ4EBVXWPC76MUYDXODPBA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.283268797 (537h55m36.154351832s) [noise=true , address=145.79.227.16:4909 , piece_id=4TSBXV4W2IFGBMZ7ILUE3XA3A5OHTDWB7KDIOXHR6MOGK5W2ZOCA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.288662038 (537h55m36.159745083s) [noise=true , address=145.79.227.19:4079 , piece_id=CDVGGB5HESIDTTYAH5VN2DH3KWDTDZH233L4YEB4ED2M7O7DG7ZQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.289184813 (537h55m36.160267848s) [noise=true , address=145.79.227.46:4601 , piece_id=GZJKDSQZ5DQYROTMBAVVRHJ7VGPWMJ6OTYNERXIAMODDMDFK77LA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.293089620 (537h55m36.164172665s) [noise=true , address=145.79.227.36:4518 , piece_id=3MGJJ3U5K4IED7BFX4SKJVRV4JB3H6ANCGDS63KICFH33H6F6YLQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.571780942 (537h55m36.442863977s) [noise=true , address=77.90.31.10:4518 , piece_id=LLDCYP7P4HOMXJBBG3QBSJTGOQORKZ23JQX64WWGROM3PKJBAM5Q, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.134061864 (537h55m37.005144909s) [noise=true , address=77.90.31.18:4808 , piece_id=Q2YDZRIECQXLO4QYZHPRNU35LGNO2R5R254X5MGACW5CIGXMVPWQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.220124597 (537h55m37.091207642s) [noise=true , address=77.90.31.13:4627 , piece_id=YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.261082414 (537h55m37.13216545s) [noise=true , address=45.91.116.5:4299 , piece_id=22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.261104707 (537h55m37.132187742s) [seg_pos={0 0}, upload=471768]: Segment reached optimal threshold of 13 pieces. Dec 28 01:24:37.261429305 (537h55m37.13251234s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:24:37.261445146 (537h55m37.132528181s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261456848 (537h55m37.132539883s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection Dec 28 01:24:37.261466185 (537h55m37.13254922s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261550937 (537h55m37.132633982s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 01:24:37.261556678 (537h55m37.132639713s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261578360 (537h55m37.132661395s) [seg_pos={0 0}, upload=471768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:24:37.261653643 (537h55m37.132736688s) [upload=471768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:24:37.261658251 (537h55m37.132741286s) [upload=471768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:24:37.543134908 (537h55m37.414217943s) [upload=471768]: Done uploading segments... Dec 28 01:24:37.543156960 (537h55m37.414240005s) [upload=471768]: Upload finished. err: <nil> Duration: 5.866001316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...XXXXX.X.XXX.X....X.X.X..XXXX....X...X..XX.X...XXX.X.XX...] duration: 6.061710144s trace: https://jaeger.dev.storj.io/trace/f8c19cb1fcf31da FAIL (2025-12-28 01:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ) context deadline exceeded; piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ) context deadline exceeded ======LOGS====== 01:24:34.012568026 (537h55m33.883651071s) [upload=471768]: Starting upload Dec 28 01:24:34.012631917 (537h55m33.883714952s) [upload=471768]: Done starting upload Dec 28 01:24:34.012746135 (537h55m33.88382917s) [upload=471768]: Uploading segments... Dec 28 01:24:34.012831286 (537h55m33.883914321s) [upload=471768]: Got next segment. Inline: false Dec 28 01:24:34.012863478 (537h55m33.883946513s) [upload=471768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:24:34.012868617 (537h55m33.883951662s) [upload=471768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:24:34.256638478 (537h55m34.127721514s) [seg_pos={0 0}, upload=471768]: Begin upload segment... Dec 28 01:24:34.256680879 (537h55m34.127763914s) [seg_pos={0 0}, upload=471768]: Done begin upload segment. Dec 28 01:24:34.256770519 (537h55m34.127853554s) [noise=true , address=145.79.227.19:4079 , piece_id=CDVGGB5HESIDTTYAH5VN2DH3KWDTDZH233L4YEB4ED2M7O7DG7ZQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.256780649 (537h55m34.127863714s) [noise=true , address=145.79.227.14:4838 , piece_id=MECTRZB3J5WC2LQHANIA7STLXZGBS2F7OMRI6OKKZYXJ45TPXAEQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.258272680 (537h55m34.129355755s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.258887171 (537h55m34.129970206s) [noise=true , address=145.79.227.16:4909 , piece_id=4TSBXV4W2IFGBMZ7ILUE3XA3A5OHTDWB7KDIOXHR6MOGK5W2ZOCA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.259331486 (537h55m34.130414531s) [noise=true , address=77.90.31.10:4518 , piece_id=LLDCYP7P4HOMXJBBG3QBSJTGOQORKZ23JQX64WWGROM3PKJBAM5Q, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.260582909 (537h55m34.131665944s) [noise=true , address=77.90.31.13:4627 , piece_id=YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.260796106 (537h55m34.131879141s) [noise=true , address=45.91.116.5:4299 , piece_id=22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.261855403 (537h55m34.132938438s) [noise=true , address=145.79.227.51:4411 , piece_id=3SREH2KDJR23ZZJWUEPUZHU2ZLIQNN4QKGJUT3O4N75GKNZOAI2A, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.262546929 (537h55m34.133629975s) [noise=true , address=145.79.227.37:4565 , piece_id=TPY3OLVJ4OJ5N6WE4WUV25UEP5TYBGZQ4EBVXWPC76MUYDXODPBA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.263149177 (537h55m34.134232212s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.264078897 (537h55m34.135161942s) [noise=true , address=145.79.227.41:4087 , piece_id=V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.264619857 (537h55m34.135702892s) [noise=true , address=145.79.227.11:4731 , piece_id=XE2GQA37LFEYBRVMG5AMN275SS7XBQ6ZMF2M43R5MJXRHODEDD6Q, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.265493841 (537h55m34.136576876s) [noise=true , address=77.90.31.18:4808 , piece_id=Q2YDZRIECQXLO4QYZHPRNU35LGNO2R5R254X5MGACW5CIGXMVPWQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.266142537 (537h55m34.137225582s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.266841297 (537h55m34.137924342s) [noise=true , address=145.79.227.36:4518 , piece_id=3MGJJ3U5K4IED7BFX4SKJVRV4JB3H6ANCGDS63KICFH33H6F6YLQ, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:34.269263941 (537h55m34.140346976s) [noise=true , address=145.79.227.46:4601 , piece_id=GZJKDSQZ5DQYROTMBAVVRHJ7VGPWMJ6OTYNERXIAMODDMDFK77LA, seg_pos={0 0}, upload=471768]: Uploading piece... Dec 28 01:24:35.889960104 (537h55m35.761043139s) [upload=471768]: Next returned nil segment Dec 28 01:24:35.889971937 (537h55m35.761054982s) [upload=471768]: Waiting for error group managing segments... Dec 28 01:24:36.275593921 (537h55m36.146676957s) [noise=true , address=145.79.227.14:4838 , piece_id=MECTRZB3J5WC2LQHANIA7STLXZGBS2F7OMRI6OKKZYXJ45TPXAEQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.277215790 (537h55m36.148298825s) [noise=true , address=145.79.227.51:4411 , piece_id=3SREH2KDJR23ZZJWUEPUZHU2ZLIQNN4QKGJUT3O4N75GKNZOAI2A, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.279432271 (537h55m36.150515306s) [noise=true , address=145.79.227.11:4731 , piece_id=XE2GQA37LFEYBRVMG5AMN275SS7XBQ6ZMF2M43R5MJXRHODEDD6Q, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.282338966 (537h55m36.153422001s) [noise=true , address=145.79.227.41:4087 , piece_id=V5OMMWXYQ6QEZ2NEBD6GAW5AEAFXKUIE2BF3XRQOCUT6E4G2U3EQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.282851623 (537h55m36.153934668s) [noise=true , address=145.79.227.37:4565 , piece_id=TPY3OLVJ4OJ5N6WE4WUV25UEP5TYBGZQ4EBVXWPC76MUYDXODPBA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.283268797 (537h55m36.154351832s) [noise=true , address=145.79.227.16:4909 , piece_id=4TSBXV4W2IFGBMZ7ILUE3XA3A5OHTDWB7KDIOXHR6MOGK5W2ZOCA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.288662038 (537h55m36.159745083s) [noise=true , address=145.79.227.19:4079 , piece_id=CDVGGB5HESIDTTYAH5VN2DH3KWDTDZH233L4YEB4ED2M7O7DG7ZQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.289184813 (537h55m36.160267848s) [noise=true , address=145.79.227.46:4601 , piece_id=GZJKDSQZ5DQYROTMBAVVRHJ7VGPWMJ6OTYNERXIAMODDMDFK77LA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.293089620 (537h55m36.164172665s) [noise=true , address=145.79.227.36:4518 , piece_id=3MGJJ3U5K4IED7BFX4SKJVRV4JB3H6ANCGDS63KICFH33H6F6YLQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:36.571780942 (537h55m36.442863977s) [noise=true , address=77.90.31.10:4518 , piece_id=LLDCYP7P4HOMXJBBG3QBSJTGOQORKZ23JQX64WWGROM3PKJBAM5Q, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.134061864 (537h55m37.005144909s) [noise=true , address=77.90.31.18:4808 , piece_id=Q2YDZRIECQXLO4QYZHPRNU35LGNO2R5R254X5MGACW5CIGXMVPWQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.220124597 (537h55m37.091207642s) [noise=true , address=77.90.31.13:4627 , piece_id=YXH4AGNWCF6YM62BVIJHF2OQACZWUDX3ZMOIR752DZGVPFIY54WA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.261082414 (537h55m37.13216545s) [noise=true , address=45.91.116.5:4299 , piece_id=22TBJQPTTZ4IENJKT3YYWWIA2DUUL7OXVMIMZJZ3PKDALIUIDOLQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: <nil> Dec 28 01:24:37.261104707 (537h55m37.132187742s) [seg_pos={0 0}, upload=471768]: Segment reached optimal threshold of 13 pieces. Dec 28 01:24:37.261429305 (537h55m37.13251234s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 01:24:37.261445146 (537h55m37.132528181s) [noise=true , address=45.91.116.4:4939 , piece_id=GMDP5HXHO2SANUEEFX2Z5Z76UJJBSFDR232RG3JW4N5QJYPBYVLA, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261456848 (537h55m37.132539883s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60710->145.79.227.4:4185: use of closed network connection Dec 28 01:24:37.261466185 (537h55m37.13254922s) [noise=true , address=145.79.227.4:4185 , piece_id=5BVNYKJROLR2G33UBNRIF2UD3XEAXV3Q4SQBDPD7P43OD4RORIVQ, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261550937 (537h55m37.132633982s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 01:24:37.261556678 (537h55m37.132639713s) [noise=true , address=77.90.31.16:4736 , piece_id=HU2DNB6TEQYL4DMD7FI6MZAMWZ2IMKQ4PUZOJ2BA4U4R3PV3DLSQ, seg_pos={0 0}, upload=471768]: Long tail cancellation due to optimal threshold Dec 28 01:24:37.261578360 (537h55m37.132661395s) [seg_pos={0 0}, upload=471768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:24:37.261653643 (537h55m37.132736688s) [upload=471768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:24:37.261658251 (537h55m37.132741286s) [upload=471768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:24:37.543134908 (537h55m37.414217943s) [upload=471768]: Done uploading segments... Dec 28 01:24:37.543156960 (537h55m37.414240005s) [upload=471768]: Upload finished. err: <nil> Duration: 6.061738658s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)