Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X..] duration: 6.107345536s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:59:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 6.107380092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........X............................................] duration: 8.072895432s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 8.072918505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X........X...........................................] duration: 22.25824774s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 22.258279451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X.] duration: 7.899838474s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:57:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 7.899875134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........X..........................................] duration: 15.367744391s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.367768908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X] duration: 30.002875049s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 30.002898574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........X.........................................] duration: 15.935156518s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:56:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.935170604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X........X........................................] duration: 56.451118318s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:56:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 56.451149737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..] duration: 8.881804071s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:55:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 8.881836221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X.] duration: 5.992913566s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:55:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 5.992944416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........X.......................................] duration: 17.96762238s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:55:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 17.967640686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........X......................................] duration: 27.284073964s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:54:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 27.284100394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X] duration: 30.002318857s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 30.002340306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........X.....................................] duration: 29.639008812s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 29.639030903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........X....................................] duration: 28.743259539s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 28.743288023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.] duration: 5.989878541s trace: https://jaeger.dev.storj.io/trace/7a8a51965de90f4d FAIL (2025-12-31 19:53:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA) context deadline exceeded ======LOGS====== 1 19:52:18.817657431 (628h23m18.688740466s) [upload=550647]: Starting upload Dec 31 19:52:18.817714298 (628h23m18.688797333s) [upload=550647]: Done starting upload Dec 31 19:52:18.817877780 (628h23m18.688960835s) [upload=550647]: Uploading segments... Dec 31 19:52:18.818458766 (628h23m18.689541801s) [upload=550647]: Got next segment. Inline: false Dec 31 19:52:18.818512238 (628h23m18.689595273s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:52:18.818534160 (628h23m18.689617195s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:52:19.073533471 (628h23m18.944616616s) [seg_pos={0 0}, upload=550647]: Begin upload segment... Dec 31 19:52:19.073628232 (628h23m18.944711307s) [seg_pos={0 0}, upload=550647]: Done begin upload segment. Dec 31 19:52:19.073663018 (628h23m18.944746053s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075170148 (628h23m18.946253193s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075272603 (628h23m18.946355638s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075791371 (628h23m18.946874416s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076129034 (628h23m18.947212139s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076173939 (628h23m18.947256984s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076500360 (628h23m18.947583395s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076756589 (628h23m18.947839634s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076887237 (628h23m18.947970272s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077319000 (628h23m18.948402035s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077344107 (628h23m18.948427152s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077729911 (628h23m18.948812946s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077880609 (628h23m18.948963684s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.078487264 (628h23m18.949570309s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079006983 (628h23m18.950090029s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079466488 (628h23m18.950549533s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:20.787238688 (628h23m20.658321733s) [upload=550647]: Next returned nil segment Dec 31 19:52:20.787251121 (628h23m20.658334156s) [upload=550647]: Waiting for error group managing segments... Dec 31 19:52:21.067516362 (628h23m20.938599407s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.086302995 (628h23m20.95738603s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.092391418 (628h23m20.963474463s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.098140284 (628h23m20.969223329s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.101875838 (628h23m20.972958883s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.105102892 (628h23m20.976185937s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.151296661 (628h23m21.022379696s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.153107429 (628h23m21.024190464s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.165848730 (628h23m21.036931765s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.261047671 (628h23m21.132130716s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.270159400 (628h23m21.141242445s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.373973044 (628h23m21.245056089s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377309598 (628h23m21.248392683s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377335407 (628h23m21.248418452s) [seg_pos={0 0}, upload=550647]: Segment reached optimal threshold of 13 pieces. Dec 31 19:52:21.377610561 (628h23m21.248693606s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377620410 (628h23m21.248703455s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377917025 (628h23m21.24900006s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection Dec 31 19:52:21.377927515 (628h23m21.24901056s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377935209 (628h23m21.249018264s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377946581 (628h23m21.249029627s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.378058975 (628h23m21.24914202s) [seg_pos={0 0}, upload=550647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:52:21.378152432 (628h23m21.249235478s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:52:21.378159447 (628h23m21.249242492s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:52:21.667390809 (628h23m21.538473855s) [upload=550647]: Done uploading segments... Dec 31 19:52:21.667415386 (628h23m21.538498421s) [upload=550647]: Upload finished. err: <nil> Duration: 5.989904771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........X...................................] duration: 25.240428965s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 25.240453713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX] duration: 30.001959663s trace: https://jaeger.dev.storj.io/trace/7a8a51965de90f4d FAIL (2025-12-31 19:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA) context deadline exceeded ======LOGS====== 1 19:52:18.817657431 (628h23m18.688740466s) [upload=550647]: Starting upload Dec 31 19:52:18.817714298 (628h23m18.688797333s) [upload=550647]: Done starting upload Dec 31 19:52:18.817877780 (628h23m18.688960835s) [upload=550647]: Uploading segments... Dec 31 19:52:18.818458766 (628h23m18.689541801s) [upload=550647]: Got next segment. Inline: false Dec 31 19:52:18.818512238 (628h23m18.689595273s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:52:18.818534160 (628h23m18.689617195s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:52:19.073533471 (628h23m18.944616616s) [seg_pos={0 0}, upload=550647]: Begin upload segment... Dec 31 19:52:19.073628232 (628h23m18.944711307s) [seg_pos={0 0}, upload=550647]: Done begin upload segment. Dec 31 19:52:19.073663018 (628h23m18.944746053s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075170148 (628h23m18.946253193s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075272603 (628h23m18.946355638s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075791371 (628h23m18.946874416s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076129034 (628h23m18.947212139s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076173939 (628h23m18.947256984s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076500360 (628h23m18.947583395s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076756589 (628h23m18.947839634s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076887237 (628h23m18.947970272s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077319000 (628h23m18.948402035s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077344107 (628h23m18.948427152s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077729911 (628h23m18.948812946s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077880609 (628h23m18.948963684s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.078487264 (628h23m18.949570309s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079006983 (628h23m18.950090029s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079466488 (628h23m18.950549533s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:20.787238688 (628h23m20.658321733s) [upload=550647]: Next returned nil segment Dec 31 19:52:20.787251121 (628h23m20.658334156s) [upload=550647]: Waiting for error group managing segments... Dec 31 19:52:21.067516362 (628h23m20.938599407s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.086302995 (628h23m20.95738603s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.092391418 (628h23m20.963474463s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.098140284 (628h23m20.969223329s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.101875838 (628h23m20.972958883s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.105102892 (628h23m20.976185937s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.151296661 (628h23m21.022379696s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.153107429 (628h23m21.024190464s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.165848730 (628h23m21.036931765s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.261047671 (628h23m21.132130716s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.270159400 (628h23m21.141242445s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.373973044 (628h23m21.245056089s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377309598 (628h23m21.248392683s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377335407 (628h23m21.248418452s) [seg_pos={0 0}, upload=550647]: Segment reached optimal threshold of 13 pieces. Dec 31 19:52:21.377610561 (628h23m21.248693606s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377620410 (628h23m21.248703455s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377917025 (628h23m21.24900006s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection Dec 31 19:52:21.377927515 (628h23m21.24901056s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377935209 (628h23m21.249018264s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377946581 (628h23m21.249029627s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.378058975 (628h23m21.24914202s) [seg_pos={0 0}, upload=550647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:52:21.378152432 (628h23m21.249235478s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:52:21.378159447 (628h23m21.249242492s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:52:21.667390809 (628h23m21.538473855s) [upload=550647]: Done uploading segments... Dec 31 19:52:21.667415386 (628h23m21.538498421s) [upload=550647]: Upload finished. err: <nil> Duration: 30.00197418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........X] duration: 30.00381083s trace: https://jaeger.dev.storj.io/trace/6c41fe23c50dbad0 FAIL (2025-12-31 19:52:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ) context deadline exceeded ======LOGS====== Dec 31 19:51:48.218191366 (628h22m48.089274401s) [upload=550639]: Starting upload Dec 31 19:51:48.218249015 (628h22m48.08933205s) [upload=550639]: Done starting upload Dec 31 19:51:48.218424068 (628h22m48.089507103s) [upload=550639]: Uploading segments... Dec 31 19:51:48.218473573 (628h22m48.089556608s) [upload=550639]: Got next segment. Inline: false Dec 31 19:51:48.218497859 (628h22m48.089580904s) [upload=550639]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:51:48.218502958 (628h22m48.089585993s) [upload=550639]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:51:48.463412829 (628h22m48.334495874s) [seg_pos={0 0}, upload=550639]: Begin upload segment... Dec 31 19:51:48.463474937 (628h22m48.334557972s) [seg_pos={0 0}, upload=550639]: Done begin upload segment. Dec 31 19:51:48.463528059 (628h22m48.334611104s) [noise=true , address=145.79.227.36:4517 , piece_id=TPETK4NVZAB7PELSEVNRIDWJMG47DULAQNWMJJD2RMFHA32WFVKQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.463533839 (628h22m48.334616874s) [noise=true , address=145.79.227.19:4079 , piece_id=VKBDOWGSA7DF5XUABZPU5D7QFJQZGLPY3BVMZCAG6C5DU7P47WTA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.463709794 (628h22m48.334792839s) [noise=true , address=145.79.227.4:4187 , piece_id=QKGVTQZS67EYQQQATPW6PHKK5Z22WGF3B4HG2GHFYBLLI2VXSJCA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464082584 (628h22m48.33516562s) [noise=true , address=45.91.116.13:4154 , piece_id=FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464448952 (628h22m48.335531997s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464585492 (628h22m48.335668537s) [noise=true , address=45.91.116.9:4264 , piece_id=REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464893007 (628h22m48.335976042s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465154035 (628h22m48.33623707s) [noise=true , address=77.90.31.11:4556 , piece_id=HJ463DL6ZUSSGY77JBJCWNCSLDQNXC6F56K72QXGRI7UTU7CIW7Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465266469 (628h22m48.336349504s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465605435 (628h22m48.33668847s) [noise=true , address=145.79.227.17:4971 , piece_id=3KVX4NU43TJ2ZGIAAKLYMFZJOIGQHRGF3JCMCJVAS7PNTHRFABKA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465626203 (628h22m48.336709249s) [noise=true , address=145.79.227.52:4447 , piece_id=7VSURSUZJOGBMRWJTVYQASRPWWFLDCYC6ZHNLNMGTWQOMX5E73MQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466033539 (628h22m48.337116584s) [noise=true , address=77.90.31.14:4663 , piece_id=FOH7IGII3MERETC6FWF2ZDMHE6VDQUYRQBF6KCSSFA4LFV3XT3PQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466039701 (628h22m48.337122736s) [noise=true , address=145.79.227.47:4266 , piece_id=Y4FNXLWJXFHX5R4BT7Q3SF5QP4OYTJM3PRVVBU2MBJ4OMGRKKI3Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466419464 (628h22m48.337502499s) [noise=true , address=145.79.227.10:4402 , piece_id=5JF5BEDDWHICPC76T3YVPOSTCJ46V73SZ4DLOLK7VRQVYHFIQEGA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466940877 (628h22m48.338023912s) [noise=true , address=145.79.227.45:4231 , piece_id=UTCXURTLGCJBE7GMVORZRTUA26ALJSZGEI4S6VXCJTSODOP2CPMQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.467401915 (628h22m48.33848496s) [noise=true , address=77.90.31.21:4878 , piece_id=57M6URN2OEZA6NTABQAHLPWU6F5ZGP2AZD4KKDAQ7ZUPNLMFE5XQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:50.170694453 (628h22m50.041777498s) [upload=550639]: Next returned nil segment Dec 31 19:51:50.170720483 (628h22m50.041803538s) [upload=550639]: Waiting for error group managing segments... Dec 31 19:51:50.454371371 (628h22m50.325454406s) [noise=true , address=145.79.227.19:4079 , piece_id=VKBDOWGSA7DF5XUABZPU5D7QFJQZGLPY3BVMZCAG6C5DU7P47WTA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.456361590 (628h22m50.327444625s) [noise=true , address=145.79.227.4:4187 , piece_id=QKGVTQZS67EYQQQATPW6PHKK5Z22WGF3B4HG2GHFYBLLI2VXSJCA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.467506983 (628h22m50.338590028s) [noise=true , address=145.79.227.52:4447 , piece_id=7VSURSUZJOGBMRWJTVYQASRPWWFLDCYC6ZHNLNMGTWQOMX5E73MQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.468201415 (628h22m50.33928446s) [noise=true , address=145.79.227.10:4402 , piece_id=5JF5BEDDWHICPC76T3YVPOSTCJ46V73SZ4DLOLK7VRQVYHFIQEGA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.474112079 (628h22m50.345195114s) [noise=true , address=145.79.227.45:4231 , piece_id=UTCXURTLGCJBE7GMVORZRTUA26ALJSZGEI4S6VXCJTSODOP2CPMQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.475382468 (628h22m50.346465503s) [noise=true , address=145.79.227.47:4266 , piece_id=Y4FNXLWJXFHX5R4BT7Q3SF5QP4OYTJM3PRVVBU2MBJ4OMGRKKI3Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.478191186 (628h22m50.349274231s) [noise=true , address=145.79.227.36:4517 , piece_id=TPETK4NVZAB7PELSEVNRIDWJMG47DULAQNWMJJD2RMFHA32WFVKQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.495099462 (628h22m50.366182497s) [noise=true , address=145.79.227.17:4971 , piece_id=3KVX4NU43TJ2ZGIAAKLYMFZJOIGQHRGF3JCMCJVAS7PNTHRFABKA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.717526340 (628h22m50.588609386s) [noise=true , address=77.90.31.11:4556 , piece_id=HJ463DL6ZUSSGY77JBJCWNCSLDQNXC6F56K72QXGRI7UTU7CIW7Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.728020995 (628h22m50.59910404s) [noise=true , address=77.90.31.14:4663 , piece_id=FOH7IGII3MERETC6FWF2ZDMHE6VDQUYRQBF6KCSSFA4LFV3XT3PQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.729282246 (628h22m50.600365291s) [noise=true , address=77.90.31.21:4878 , piece_id=57M6URN2OEZA6NTABQAHLPWU6F5ZGP2AZD4KKDAQ7ZUPNLMFE5XQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.782546027 (628h22m50.653629062s) [noise=true , address=45.91.116.9:4264 , piece_id=REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.847626249 (628h22m50.718709284s) [noise=true , address=45.91.116.13:4154 , piece_id=FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.847645876 (628h22m50.718728921s) [seg_pos={0 0}, upload=550639]: Segment reached optimal threshold of 13 pieces. Dec 31 19:51:50.847818755 (628h22m50.71890179s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.847830328 (628h22m50.718913363s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.847940657 (628h22m50.719023703s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.847948693 (628h22m50.719031728s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.848089400 (628h22m50.719172435s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.848095191 (628h22m50.719178236s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.848110932 (628h22m50.719193967s) [seg_pos={0 0}, upload=550639]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:51:50.848189501 (628h22m50.719272546s) [upload=550639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:51:50.848194971 (628h22m50.719278006s) [upload=550639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:51:51.137679055 (628h22m51.00876209s) [upload=550639]: Done uploading segments... Dec 31 19:51:51.137700256 (628h22m51.008783291s) [upload=550639]: Upload finished. err: <nil> Duration: 30.003837841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........X..................................] duration: 27.452848552s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 27.452877086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........] duration: 6.169205659s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 6.169226227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........X.................................] duration: 15.674994053s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.675024491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........X................................] duration: 21.942208453s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 21.942234572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X........X...............................] duration: 25.750803735s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 25.750830697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X........] duration: 5.826148068s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 5.826181692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.......] duration: 5.97241475s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:49:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 5.972466037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X........X..............................] duration: 29.192548233s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:49:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 29.192576546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X......] duration: 6.465338776s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:49:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 6.465416676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.....] duration: 5.778227984s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 5.778275496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X........X.............................] duration: 22.280221652s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 22.280254064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X........X............................] duration: 32.435266117s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 32.43529968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X....] duration: 6.151653855s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:47:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 6.151670268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X...] duration: 5.914275145s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:47:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 5.914298902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X........X...........................] duration: 21.51610668s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 21.516138029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X..] duration: 5.933352282s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 5.933377219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X........X..........................] duration: 35.84393125s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 35.84394751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.] duration: 9.290349894s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 9.290365054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X........X.........................] duration: 35.641472716s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:46:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 35.641506511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X] duration: 30.004042733s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:46:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 30.004071998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.] duration: 5.970054736s trace: https://jaeger.dev.storj.io/trace/adc0f1bde45046b FAIL (2025-12-31 19:45:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A) context deadline exceeded ======LOGS====== Dec 31 19:44:28.559831708 (628h15m28.430914743s) [upload=550526]: Starting upload Dec 31 19:44:28.559883857 (628h15m28.430966892s) [upload=550526]: Done starting upload Dec 31 19:44:28.560082255 (628h15m28.43116529s) [upload=550526]: Uploading segments... Dec 31 19:44:28.560122411 (628h15m28.431205446s) [upload=550526]: Got next segment. Inline: false Dec 31 19:44:28.560147619 (628h15m28.431230665s) [upload=550526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:44:28.560151487 (628h15m28.431234522s) [upload=550526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:44:28.810061068 (628h15m28.681144103s) [seg_pos={0 0}, upload=550526]: Begin upload segment... Dec 31 19:44:28.810143574 (628h15m28.681226609s) [seg_pos={0 0}, upload=550526]: Done begin upload segment. Dec 31 19:44:28.810230821 (628h15m28.681313866s) [noise=true , address=45.91.116.14:4189 , piece_id=LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810273742 (628h15m28.681356788s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810335971 (628h15m28.681419016s) [noise=true , address=145.79.227.23:4546 , piece_id=6NRO5KKFQ2LJ4RVOPL5H46EAPYFUGQRWMCUCLE7T5ZP6YSZRPSCQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810925754 (628h15m28.682008799s) [noise=true , address=145.79.227.11:4729 , piece_id=DBXOXPGMWTNGVY3NXLDIGKBISGOWFESGDSIEMNJRUXGL7SLTHWMQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811123400 (628h15m28.682206445s) [noise=true , address=145.79.227.12:4765 , piece_id=INLTE6B7R6BVJ2VPVMZHF3JQAH5K4KW2VQJPPMSFXCDABL6ZQWXQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811546345 (628h15m28.68262938s) [noise=true , address=45.91.116.11:4120 , piece_id=6JW6J47IK6EYSPHZNGF6WAQGTOMSCJA5O22EBNAARUONWMB6UFFQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811645975 (628h15m28.68272902s) [noise=true , address=45.91.116.13:4153 , piece_id=DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812037812 (628h15m28.683120857s) [noise=true , address=145.79.227.7:4294 , piece_id=XG5K3Q7KGHEE5DZYKMMJMC4CXSY3MJQOP3KQQTDXR3IJHSH7ZSXA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812214248 (628h15m28.683297293s) [noise=true , address=45.91.116.9:4262 , piece_id=4CIQCW4HMLH23JKMVHEOZW3G3XIUNZVVRRUPTVRV7P5G4JGHAYWA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812794833 (628h15m28.683877878s) [noise=true , address=77.90.31.13:4629 , piece_id=E6KKRLSSBLRPME3SIBM2Y4U4CRVZP466XMCKSPR45MWLI4QT2ZPQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.813315294 (628h15m28.684398339s) [noise=true , address=145.79.227.50:4374 , piece_id=TJJHLRBBNYQB7BL5IKIMY7MPYEABTE4FJKE6DSV2UHEKZUZQQ5WA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.813773356 (628h15m28.684856391s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814322973 (628h15m28.685406018s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814563941 (628h15m28.685646977s) [noise=true , address=145.79.227.51:4410 , piece_id=PBNOTDA5LG2Q3L2SPIJMUH2W5T3EFUWLWOXB3FBWSASAAMEDZPRA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814959264 (628h15m28.686042319s) [noise=true , address=145.79.227.5:4221 , piece_id=OS7X2KGQ6IC775DEEZSC2PUIV4ZUTHIYCFDHBEL6PHIUMCYUMVEQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.815423789 (628h15m28.686506824s) [noise=true , address=145.79.227.48:4301 , piece_id=ARRBMIM3BX32H6TUJAUJO4DGWU5SOBAFZN22LFAFZP3HVPNYYJ6Q, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:30.473832310 (628h15m30.344915345s) [upload=550526]: Next returned nil segment Dec 31 19:44:30.473855724 (628h15m30.344938759s) [upload=550526]: Waiting for error group managing segments... Dec 31 19:44:30.807360562 (628h15m30.678443597s) [noise=true , address=145.79.227.5:4221 , piece_id=OS7X2KGQ6IC775DEEZSC2PUIV4ZUTHIYCFDHBEL6PHIUMCYUMVEQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.814539411 (628h15m30.685622446s) [noise=true , address=145.79.227.51:4410 , piece_id=PBNOTDA5LG2Q3L2SPIJMUH2W5T3EFUWLWOXB3FBWSASAAMEDZPRA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.822579841 (628h15m30.693662876s) [noise=true , address=145.79.227.23:4546 , piece_id=6NRO5KKFQ2LJ4RVOPL5H46EAPYFUGQRWMCUCLE7T5ZP6YSZRPSCQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.825177578 (628h15m30.696260613s) [noise=true , address=145.79.227.50:4374 , piece_id=TJJHLRBBNYQB7BL5IKIMY7MPYEABTE4FJKE6DSV2UHEKZUZQQ5WA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.887571003 (628h15m30.758654038s) [noise=true , address=145.79.227.12:4765 , piece_id=INLTE6B7R6BVJ2VPVMZHF3JQAH5K4KW2VQJPPMSFXCDABL6ZQWXQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.889848620 (628h15m30.760931655s) [noise=true , address=145.79.227.7:4294 , piece_id=XG5K3Q7KGHEE5DZYKMMJMC4CXSY3MJQOP3KQQTDXR3IJHSH7ZSXA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.900591465 (628h15m30.77167451s) [noise=true , address=145.79.227.48:4301 , piece_id=ARRBMIM3BX32H6TUJAUJO4DGWU5SOBAFZN22LFAFZP3HVPNYYJ6Q, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.949223086 (628h15m30.820306121s) [noise=true , address=145.79.227.11:4729 , piece_id=DBXOXPGMWTNGVY3NXLDIGKBISGOWFESGDSIEMNJRUXGL7SLTHWMQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.967088926 (628h15m30.838171961s) [noise=true , address=77.90.31.13:4629 , piece_id=E6KKRLSSBLRPME3SIBM2Y4U4CRVZP466XMCKSPR45MWLI4QT2ZPQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.109310735 (628h15m30.98039378s) [noise=true , address=45.91.116.14:4189 , piece_id=LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.129863389 (628h15m31.000946435s) [noise=true , address=45.91.116.9:4262 , piece_id=4CIQCW4HMLH23JKMVHEOZW3G3XIUNZVVRRUPTVRV7P5G4JGHAYWA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.139036656 (628h15m31.010119701s) [noise=true , address=45.91.116.13:4153 , piece_id=DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.148420445 (628h15m31.01950348s) [noise=true , address=45.91.116.11:4120 , piece_id=6JW6J47IK6EYSPHZNGF6WAQGTOMSCJA5O22EBNAARUONWMB6UFFQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.148441655 (628h15m31.01952469s) [seg_pos={0 0}, upload=550526]: Segment reached optimal threshold of 13 pieces. Dec 31 19:44:31.148627660 (628h15m31.019710705s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.148634382 (628h15m31.019717428s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.148903364 (628h15m31.019986409s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.148913965 (628h15m31.01999701s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.149180632 (628h15m31.020263667s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.149188187 (628h15m31.020271232s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.149203897 (628h15m31.020286942s) [seg_pos={0 0}, upload=550526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:44:31.149273449 (628h15m31.020356494s) [upload=550526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:44:31.149278849 (628h15m31.020361894s) [upload=550526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:44:31.438713561 (628h15m31.309796596s) [upload=550526]: Done uploading segments... Dec 31 19:44:31.438738399 (628h15m31.309821434s) [upload=550526]: Upload finished. err: <nil> Duration: 5.970074042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X] duration: 30.002700571s trace: https://jaeger.dev.storj.io/trace/adc0f1bde45046b FAIL (2025-12-31 19:44:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A) context deadline exceeded ======LOGS====== Dec 31 19:44:28.559831708 (628h15m28.430914743s) [upload=550526]: Starting upload Dec 31 19:44:28.559883857 (628h15m28.430966892s) [upload=550526]: Done starting upload Dec 31 19:44:28.560082255 (628h15m28.43116529s) [upload=550526]: Uploading segments... Dec 31 19:44:28.560122411 (628h15m28.431205446s) [upload=550526]: Got next segment. Inline: false Dec 31 19:44:28.560147619 (628h15m28.431230665s) [upload=550526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:44:28.560151487 (628h15m28.431234522s) [upload=550526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:44:28.810061068 (628h15m28.681144103s) [seg_pos={0 0}, upload=550526]: Begin upload segment... Dec 31 19:44:28.810143574 (628h15m28.681226609s) [seg_pos={0 0}, upload=550526]: Done begin upload segment. Dec 31 19:44:28.810230821 (628h15m28.681313866s) [noise=true , address=45.91.116.14:4189 , piece_id=LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810273742 (628h15m28.681356788s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810335971 (628h15m28.681419016s) [noise=true , address=145.79.227.23:4546 , piece_id=6NRO5KKFQ2LJ4RVOPL5H46EAPYFUGQRWMCUCLE7T5ZP6YSZRPSCQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.810925754 (628h15m28.682008799s) [noise=true , address=145.79.227.11:4729 , piece_id=DBXOXPGMWTNGVY3NXLDIGKBISGOWFESGDSIEMNJRUXGL7SLTHWMQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811123400 (628h15m28.682206445s) [noise=true , address=145.79.227.12:4765 , piece_id=INLTE6B7R6BVJ2VPVMZHF3JQAH5K4KW2VQJPPMSFXCDABL6ZQWXQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811546345 (628h15m28.68262938s) [noise=true , address=45.91.116.11:4120 , piece_id=6JW6J47IK6EYSPHZNGF6WAQGTOMSCJA5O22EBNAARUONWMB6UFFQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.811645975 (628h15m28.68272902s) [noise=true , address=45.91.116.13:4153 , piece_id=DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812037812 (628h15m28.683120857s) [noise=true , address=145.79.227.7:4294 , piece_id=XG5K3Q7KGHEE5DZYKMMJMC4CXSY3MJQOP3KQQTDXR3IJHSH7ZSXA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812214248 (628h15m28.683297293s) [noise=true , address=45.91.116.9:4262 , piece_id=4CIQCW4HMLH23JKMVHEOZW3G3XIUNZVVRRUPTVRV7P5G4JGHAYWA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.812794833 (628h15m28.683877878s) [noise=true , address=77.90.31.13:4629 , piece_id=E6KKRLSSBLRPME3SIBM2Y4U4CRVZP466XMCKSPR45MWLI4QT2ZPQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.813315294 (628h15m28.684398339s) [noise=true , address=145.79.227.50:4374 , piece_id=TJJHLRBBNYQB7BL5IKIMY7MPYEABTE4FJKE6DSV2UHEKZUZQQ5WA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.813773356 (628h15m28.684856391s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814322973 (628h15m28.685406018s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814563941 (628h15m28.685646977s) [noise=true , address=145.79.227.51:4410 , piece_id=PBNOTDA5LG2Q3L2SPIJMUH2W5T3EFUWLWOXB3FBWSASAAMEDZPRA, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.814959264 (628h15m28.686042319s) [noise=true , address=145.79.227.5:4221 , piece_id=OS7X2KGQ6IC775DEEZSC2PUIV4ZUTHIYCFDHBEL6PHIUMCYUMVEQ, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:28.815423789 (628h15m28.686506824s) [noise=true , address=145.79.227.48:4301 , piece_id=ARRBMIM3BX32H6TUJAUJO4DGWU5SOBAFZN22LFAFZP3HVPNYYJ6Q, seg_pos={0 0}, upload=550526]: Uploading piece... Dec 31 19:44:30.473832310 (628h15m30.344915345s) [upload=550526]: Next returned nil segment Dec 31 19:44:30.473855724 (628h15m30.344938759s) [upload=550526]: Waiting for error group managing segments... Dec 31 19:44:30.807360562 (628h15m30.678443597s) [noise=true , address=145.79.227.5:4221 , piece_id=OS7X2KGQ6IC775DEEZSC2PUIV4ZUTHIYCFDHBEL6PHIUMCYUMVEQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.814539411 (628h15m30.685622446s) [noise=true , address=145.79.227.51:4410 , piece_id=PBNOTDA5LG2Q3L2SPIJMUH2W5T3EFUWLWOXB3FBWSASAAMEDZPRA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.822579841 (628h15m30.693662876s) [noise=true , address=145.79.227.23:4546 , piece_id=6NRO5KKFQ2LJ4RVOPL5H46EAPYFUGQRWMCUCLE7T5ZP6YSZRPSCQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.825177578 (628h15m30.696260613s) [noise=true , address=145.79.227.50:4374 , piece_id=TJJHLRBBNYQB7BL5IKIMY7MPYEABTE4FJKE6DSV2UHEKZUZQQ5WA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.887571003 (628h15m30.758654038s) [noise=true , address=145.79.227.12:4765 , piece_id=INLTE6B7R6BVJ2VPVMZHF3JQAH5K4KW2VQJPPMSFXCDABL6ZQWXQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.889848620 (628h15m30.760931655s) [noise=true , address=145.79.227.7:4294 , piece_id=XG5K3Q7KGHEE5DZYKMMJMC4CXSY3MJQOP3KQQTDXR3IJHSH7ZSXA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.900591465 (628h15m30.77167451s) [noise=true , address=145.79.227.48:4301 , piece_id=ARRBMIM3BX32H6TUJAUJO4DGWU5SOBAFZN22LFAFZP3HVPNYYJ6Q, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.949223086 (628h15m30.820306121s) [noise=true , address=145.79.227.11:4729 , piece_id=DBXOXPGMWTNGVY3NXLDIGKBISGOWFESGDSIEMNJRUXGL7SLTHWMQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:30.967088926 (628h15m30.838171961s) [noise=true , address=77.90.31.13:4629 , piece_id=E6KKRLSSBLRPME3SIBM2Y4U4CRVZP466XMCKSPR45MWLI4QT2ZPQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.109310735 (628h15m30.98039378s) [noise=true , address=45.91.116.14:4189 , piece_id=LSSW3CNVGIL4KM2MF2LM4SFUNZH33T42VH6EUCSIOD7YJAREZXEQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.129863389 (628h15m31.000946435s) [noise=true , address=45.91.116.9:4262 , piece_id=4CIQCW4HMLH23JKMVHEOZW3G3XIUNZVVRRUPTVRV7P5G4JGHAYWA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.139036656 (628h15m31.010119701s) [noise=true , address=45.91.116.13:4153 , piece_id=DCA56HUYMN3QZCMHEWTZHLS3AABZCC4KAHCSBSJD5VDSBSJWAJ6A, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.148420445 (628h15m31.01950348s) [noise=true , address=45.91.116.11:4120 , piece_id=6JW6J47IK6EYSPHZNGF6WAQGTOMSCJA5O22EBNAARUONWMB6UFFQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: <nil> Dec 31 19:44:31.148441655 (628h15m31.01952469s) [seg_pos={0 0}, upload=550526]: Segment reached optimal threshold of 13 pieces. Dec 31 19:44:31.148627660 (628h15m31.019710705s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.148634382 (628h15m31.019717428s) [noise=true , address=45.91.116.12:4225 , piece_id=4S6F6WB4ML6ETB43HFXZKE6IO7YR2AIDHAVPEZFKMJ6XO7EL73JQ, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.148903364 (628h15m31.019986409s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.148913965 (628h15m31.01999701s) [noise=true , address=77.90.31.16:4736 , piece_id=VXEU4DOR7V7NNILPHFRGFLCTRAWO6W4QN7C46THMO4E35DNGDCLQ, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.149180632 (628h15m31.020263667s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 19:44:31.149188187 (628h15m31.020271232s) [noise=true , address=145.79.227.30:4762 , piece_id=H6226M3YPQ2UKIYJFUXQCII2NHOC5JQXJRX6KAYZPM6IMSKSRNVA, seg_pos={0 0}, upload=550526]: Long tail cancellation due to optimal threshold Dec 31 19:44:31.149203897 (628h15m31.020286942s) [seg_pos={0 0}, upload=550526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:44:31.149273449 (628h15m31.020356494s) [upload=550526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:44:31.149278849 (628h15m31.020361894s) [upload=550526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:44:31.438713561 (628h15m31.309796596s) [upload=550526]: Done uploading segments... Dec 31 19:44:31.438738399 (628h15m31.309821434s) [upload=550526]: Upload finished. err: <nil> Duration: 30.002718555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X........X........................] duration: 26.093516868s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 26.093545942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X........X.......................] duration: 18.350395873s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 18.350414588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..] duration: 27.417116416s trace: https://jaeger.dev.storj.io/trace/218eac638091a67d FAIL (2025-12-31 19:44:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Dec 31 19:43:10.417983930 (628h14m10.289066975s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:43:10.417992537 (628h14m10.289075582s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:43:10.669916273 (628h14m10.540999308s) [seg_pos={0 0}, upload=550501]: Begin upload segment... Dec 31 19:43:10.669959266 (628h14m10.541042301s) [seg_pos={0 0}, upload=550501]: Done begin upload segment. Dec 31 19:43:10.669990003 (628h14m10.541073048s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670000935 (628h14m10.54108398s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670387100 (628h14m10.541470135s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670646664 (628h14m10.541729699s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671043330 (628h14m10.542126365s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671343361 (628h14m10.542426406s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671607123 (628h14m10.542690168s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671911653 (628h14m10.542994688s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.672286055 (628h14m10.543369091s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.673985051 (628h14m10.545068096s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.674041619 (628h14m10.545124664s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.675105945 (628h14m10.54618898s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.676208144 (628h14m10.547291179s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.677266058 (628h14m10.548349093s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.678305948 (628h14m10.549388984s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.682521245 (628h14m10.55360429s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:12.373875254 (628h14m12.244958299s) [upload=550501]: Next returned nil segment Dec 31 19:43:12.373899100 (628h14m12.244982145s) [upload=550501]: Waiting for error group managing segments... Dec 31 19:43:12.657643096 (628h14m12.528726131s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.658192873 (628h14m12.529275908s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.661570093 (628h14m12.532653128s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.664843246 (628h14m12.535926281s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.671463001 (628h14m12.542546046s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.686567773 (628h14m12.557650808s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.689763026 (628h14m12.560846062s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.690429796 (628h14m12.561512831s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.691414171 (628h14m12.562497226s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.692946127 (628h14m12.564029182s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.710688452 (628h14m12.581771497s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.945231964 (628h14m12.816315009s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069110765 (628h14m12.94019381s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069131123 (628h14m12.940214168s) [seg_pos={0 0}, upload=550501]: Segment reached optimal threshold of 13 pieces. Dec 31 19:43:13.069355571 (628h14m12.940438606s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection Dec 31 19:43:13.069365039 (628h14m12.940448084s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069374376 (628h14m12.940457411s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:45548->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:43:13.069382943 (628h14m12.940465978s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069472343 (628h14m12.940555378s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection Dec 31 19:43:13.069477462 (628h14m12.940560507s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069489276 (628h14m12.940572321s) [seg_pos={0 0}, upload=550501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:43:13.069532568 (628h14m12.940615603s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:43:13.069536335 (628h14m12.94061937s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:43:13.359386667 (628h14m13.230469702s) [upload=550501]: Done uploading segments... Dec 31 19:43:13.359403800 (628h14m13.230486835s) [upload=550501]: Upload finished. err: <nil> Duration: 27.417154037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX.] duration: 6.205163987s trace: https://jaeger.dev.storj.io/trace/218eac638091a67d FAIL (2025-12-31 19:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Dec 31 19:43:10.417983930 (628h14m10.289066975s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:43:10.417992537 (628h14m10.289075582s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:43:10.669916273 (628h14m10.540999308s) [seg_pos={0 0}, upload=550501]: Begin upload segment... Dec 31 19:43:10.669959266 (628h14m10.541042301s) [seg_pos={0 0}, upload=550501]: Done begin upload segment. Dec 31 19:43:10.669990003 (628h14m10.541073048s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670000935 (628h14m10.54108398s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670387100 (628h14m10.541470135s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670646664 (628h14m10.541729699s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671043330 (628h14m10.542126365s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671343361 (628h14m10.542426406s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671607123 (628h14m10.542690168s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671911653 (628h14m10.542994688s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.672286055 (628h14m10.543369091s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.673985051 (628h14m10.545068096s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.674041619 (628h14m10.545124664s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.675105945 (628h14m10.54618898s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.676208144 (628h14m10.547291179s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.677266058 (628h14m10.548349093s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.678305948 (628h14m10.549388984s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.682521245 (628h14m10.55360429s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:12.373875254 (628h14m12.244958299s) [upload=550501]: Next returned nil segment Dec 31 19:43:12.373899100 (628h14m12.244982145s) [upload=550501]: Waiting for error group managing segments... Dec 31 19:43:12.657643096 (628h14m12.528726131s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.658192873 (628h14m12.529275908s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.661570093 (628h14m12.532653128s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.664843246 (628h14m12.535926281s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.671463001 (628h14m12.542546046s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.686567773 (628h14m12.557650808s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.689763026 (628h14m12.560846062s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.690429796 (628h14m12.561512831s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.691414171 (628h14m12.562497226s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.692946127 (628h14m12.564029182s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.710688452 (628h14m12.581771497s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.945231964 (628h14m12.816315009s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069110765 (628h14m12.94019381s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069131123 (628h14m12.940214168s) [seg_pos={0 0}, upload=550501]: Segment reached optimal threshold of 13 pieces. Dec 31 19:43:13.069355571 (628h14m12.940438606s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection Dec 31 19:43:13.069365039 (628h14m12.940448084s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069374376 (628h14m12.940457411s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:45548->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:43:13.069382943 (628h14m12.940465978s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069472343 (628h14m12.940555378s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection Dec 31 19:43:13.069477462 (628h14m12.940560507s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069489276 (628h14m12.940572321s) [seg_pos={0 0}, upload=550501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:43:13.069532568 (628h14m12.940615603s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:43:13.069536335 (628h14m12.94061937s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:43:13.359386667 (628h14m13.230469702s) [upload=550501]: Done uploading segments... Dec 31 19:43:13.359403800 (628h14m13.230486835s) [upload=550501]: Upload finished. err: <nil> Duration: 6.205179286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX] duration: 30.002428895s trace: https://jaeger.dev.storj.io/trace/218eac638091a67d FAIL (2025-12-31 19:43:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== false Dec 31 19:43:10.417983930 (628h14m10.289066975s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:43:10.417992537 (628h14m10.289075582s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:43:10.669916273 (628h14m10.540999308s) [seg_pos={0 0}, upload=550501]: Begin upload segment... Dec 31 19:43:10.669959266 (628h14m10.541042301s) [seg_pos={0 0}, upload=550501]: Done begin upload segment. Dec 31 19:43:10.669990003 (628h14m10.541073048s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670000935 (628h14m10.54108398s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670387100 (628h14m10.541470135s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.670646664 (628h14m10.541729699s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671043330 (628h14m10.542126365s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671343361 (628h14m10.542426406s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671607123 (628h14m10.542690168s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.671911653 (628h14m10.542994688s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.672286055 (628h14m10.543369091s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.673985051 (628h14m10.545068096s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.674041619 (628h14m10.545124664s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.675105945 (628h14m10.54618898s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.676208144 (628h14m10.547291179s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.677266058 (628h14m10.548349093s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.678305948 (628h14m10.549388984s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:10.682521245 (628h14m10.55360429s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Uploading piece... Dec 31 19:43:12.373875254 (628h14m12.244958299s) [upload=550501]: Next returned nil segment Dec 31 19:43:12.373899100 (628h14m12.244982145s) [upload=550501]: Waiting for error group managing segments... Dec 31 19:43:12.657643096 (628h14m12.528726131s) [noise=true , address=145.79.227.37:4567 , piece_id=TEQL6QM32FQUOKWI7NMALM3RT4IDEOS4YX6IIR4H7TRBKPZ4BEHA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.658192873 (628h14m12.529275908s) [noise=true , address=145.79.227.2:4115 , piece_id=WWGYUDEO4LWIJ44N6S56W5NKUOOVTZLKNCA6BOI5U2OPOZFCIFIA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.661570093 (628h14m12.532653128s) [noise=true , address=145.79.227.5:4223 , piece_id=RR7WZYIY6J6U5PSCVEAQ65SBLFIHFW3NI3W4C76WEIFZHMKPRG3Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.664843246 (628h14m12.535926281s) [noise=true , address=145.79.227.10:4401 , piece_id=QVPD7FNX5CO4ZFWL6RFMB22PAGKDD7V7LF353SB4NRIDLOO3ZYZQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.671463001 (628h14m12.542546046s) [noise=true , address=145.79.227.27:4691 , piece_id=GYMVCHYQAM7E5F6QFAIJ5VIJ22EIXEZKW7MVP4YPAOX6WRFK3I5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.686567773 (628h14m12.557650808s) [noise=true , address=145.79.227.12:4767 , piece_id=3AKMWLXSYGKLHP4ZHTGUGQ5CI7STICNE2PBOWLOMZUJIKVAJ3BQA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.689763026 (628h14m12.560846062s) [noise=true , address=145.79.227.3:4149 , piece_id=Y2RFLZWGEMQ2K3ZRCDFTU6BISXOSGKXCHV2ODZ7T2HE5EVIMVTOA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.690429796 (628h14m12.561512831s) [noise=true , address=145.79.227.47:4266 , piece_id=PI3CV25PATQMJLBFKJPC6XSMXSIBNUH3SN4RRTK2XEKEBT6G7BDQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.691414171 (628h14m12.562497226s) [noise=true , address=145.79.227.14:4838 , piece_id=333JGSD6BTTW62T5BTW22Y6I5ODVKMSIIKMO2TO32BE743JLQ43A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.692946127 (628h14m12.564029182s) [noise=true , address=145.79.227.22:4511 , piece_id=BOSLNOKIUZ5FNKXY4CFFYSYSNXFPFUUJAE7LEUQNZT5YA6QMAIYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.710688452 (628h14m12.581771497s) [noise=true , address=145.79.227.16:4909 , piece_id=VL3I572EAWGWEO2THMBT2XGHQNGQNV2IJJV6GQN6BVN3J7XEJE5Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:12.945231964 (628h14m12.816315009s) [noise=true , address=77.90.31.21:4880 , piece_id=JAWJWIEMKLN2RBE5E2Y47CLCNFNLSLWUGIVFSWNFPVHXG33IQLIQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069110765 (628h14m12.94019381s) [noise=true , address=45.91.116.10:4083 , piece_id=YIEUSA5YDWGHKW7GQLQZQBRGFUYCCOKD36ASA4TOAGXMFUWSCX4Q, seg_pos={0 0}, upload=550501]: Done uploading piece. err: <nil> Dec 31 19:43:13.069131123 (628h14m12.940214168s) [seg_pos={0 0}, upload=550501]: Segment reached optimal threshold of 13 pieces. Dec 31 19:43:13.069355571 (628h14m12.940438606s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59690->45.91.116.6:4975: use of closed network connection Dec 31 19:43:13.069365039 (628h14m12.940448084s) [noise=true , address=45.91.116.6:4975 , piece_id=RTWXCLLPUQGMOHQ3ALT6G53U6YV7NHNX3LCERYIGH3VCAZJ2BCYQ, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069374376 (628h14m12.940457411s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:45548->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:43:13.069382943 (628h14m12.940465978s) [noise=true , address=45.91.116.8:4046 , piece_id=AIYFJYMZQ2R4HC5643SHWFFC2BJLS36CAJRTWTHZTUUA7I4BHQEA, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069472343 (628h14m12.940555378s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60626->77.90.31.19:4841: use of closed network connection Dec 31 19:43:13.069477462 (628h14m12.940560507s) [noise=true , address=77.90.31.19:4841 , piece_id=BZPPGBTYL7UJZSVWRBHPAN74M6G757LRHOT4U44Y2O5NPDR4JB3A, seg_pos={0 0}, upload=550501]: Long tail cancellation due to optimal threshold Dec 31 19:43:13.069489276 (628h14m12.940572321s) [seg_pos={0 0}, upload=550501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:43:13.069532568 (628h14m12.940615603s) [upload=550501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:43:13.069536335 (628h14m12.94061937s) [upload=550501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:43:13.359386667 (628h14m13.230469702s) [upload=550501]: Done uploading segments... Dec 31 19:43:13.359403800 (628h14m13.230486835s) [upload=550501]: Upload finished. err: <nil> Duration: 30.002452911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X........X......................] duration: 15.350580044s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:43:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.350608679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......X] duration: 30.002590462s trace: https://jaeger.dev.storj.io/trace/2eac40da5ae1d77e FAIL (2025-12-31 19:42:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:42:05.754622494 (628h13m5.625705529s) [upload=550486]: Starting upload Dec 31 19:42:05.754686676 (628h13m5.625769711s) [upload=550486]: Done starting upload Dec 31 19:42:05.754727333 (628h13m5.625810378s) [upload=550486]: Uploading segments... Dec 31 19:42:05.754972460 (628h13m5.626055505s) [upload=550486]: Got next segment. Inline: false Dec 31 19:42:05.755015432 (628h13m5.626098477s) [upload=550486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:42:05.755021844 (628h13m5.626104889s) [upload=550486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:42:06.000890361 (628h13m5.871973426s) [seg_pos={0 0}, upload=550486]: Begin upload segment... Dec 31 19:42:06.000960605 (628h13m5.87204365s) [seg_pos={0 0}, upload=550486]: Done begin upload segment. Dec 31 19:42:06.000979330 (628h13m5.872062365s) [noise=true , address=145.79.227.8:4331 , piece_id=3KDRQDN4AWU3YYCQKLBNJE3CWZSZTQY6JIB6SHV5SRVGGZJNSVWA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.001001763 (628h13m5.872084828s) [noise=true , address=77.90.31.16:4735 , piece_id=NL4PZIASTKCFBF3J3CMVAFUOJDRAENWKMZ5H4TGJZ6MZ5S472X6Q, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.002057272 (628h13m5.873140317s) [noise=true , address=77.90.31.21:4877 , piece_id=DSPYWJ7BQ7JYALHQLSMKYG2WZJHRFCTIQAAFDH57SLDN6LIX4BCQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.002521787 (628h13m5.873604833s) [noise=true , address=45.91.116.7:4010 , piece_id=AYPDZW73FOAIIGRDWRZ562KW3TB33ZS5OC6UWV5RZ35T5WQBORFA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.002930515 (628h13m5.87401356s) [noise=true , address=145.79.227.5:4223 , piece_id=47UQJFE3KXIZ7UBPRST3BWDGDJLIVAG6OUUGC5YB64VKZHQ4M5MQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.003327962 (628h13m5.874411007s) [noise=true , address=145.79.227.24:4583 , piece_id=AESUXVKBYKDYDKJG7EK76UKRTYT44N2CZWEMGSU6HDLS5UAHBWEQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.003784592 (628h13m5.874867637s) [noise=true , address=145.79.227.10:4401 , piece_id=ZRR7YLF2YZGYN4GON7HRWHHV23OYI5WYA7TUYZTWVZUKJWRHETNQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.003791966 (628h13m5.874875011s) [noise=true , address=45.91.116.5:4298 , piece_id=WECEEDNWTD7WDJQMPE3BH6X4QPT474E2XG4SOYYYMQ5DY7XMY3VQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.004212897 (628h13m5.875295942s) [noise=true , address=45.91.116.4:4940 , piece_id=DXP7SAMSFQQK7ZY6K5X7ADMBXGGY5UGTEODB4CE2ZQDZEMJRQPUA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.004362051 (628h13m5.875445096s) [noise=true , address=145.79.227.2:4114 , piece_id=A2AMMEDUDQGL7FW47MYKS25YHSB3M4EAZUJU7XA67QS3HEUCC45A, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.004790728 (628h13m5.875873763s) [noise=true , address=77.90.31.19:4841 , piece_id=WPK5YBWYJWKP6GDL4DXMWSCQGT5DHOKPQIHLQQIEK2GBRBYYPMQA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.011392578 (628h13m5.882475633s) [noise=true , address=145.79.227.50:4374 , piece_id=FQ5YIJOGQASFPFA6M7UPCLL4TKUEDB6WUTP3G7GZRRVMUQXBL3VA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.018500453 (628h13m5.889583498s) [noise=true , address=145.79.227.6:4257 , piece_id=BTHNAJ35EWXHKFHGOLWOIJKU2RVT5UNVTEJS5EEWWENCTKV2H2JQ, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.021395375 (628h13m5.89247842s) [noise=true , address=145.79.227.15:4876 , piece_id=EP55WINK3OOLCR2AW7HA4XSCKAY3NT7TR2QQ5QU4Y3XIN6OXU6FA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.022147037 (628h13m5.893230092s) [noise=true , address=45.91.116.10:4083 , piece_id=OTIBS53ZZKEYSLZWVDM6VT6D7LLVTVRFIBCQCPQ2IIV6IJWYSEPA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:06.024288604 (628h13m5.895371639s) [noise=true , address=145.79.227.32:4835 , piece_id=4R6B2WBQWI5P6TXONXOFUQFSJZODOOKQLOE4ANQ5FZX7J5KP3QXA, seg_pos={0 0}, upload=550486]: Uploading piece... Dec 31 19:42:07.702624388 (628h13m7.573707433s) [upload=550486]: Next returned nil segment Dec 31 19:42:07.702663654 (628h13m7.573746689s) [upload=550486]: Waiting for error group managing segments... Dec 31 19:42:07.980360164 (628h13m7.851443199s) [noise=true , address=145.79.227.5:4223 , piece_id=47UQJFE3KXIZ7UBPRST3BWDGDJLIVAG6OUUGC5YB64VKZHQ4M5MQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:07.991893125 (628h13m7.86297616s) [noise=true , address=145.79.227.6:4257 , piece_id=BTHNAJ35EWXHKFHGOLWOIJKU2RVT5UNVTEJS5EEWWENCTKV2H2JQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:07.993467943 (628h13m7.864550988s) [noise=true , address=145.79.227.8:4331 , piece_id=3KDRQDN4AWU3YYCQKLBNJE3CWZSZTQY6JIB6SHV5SRVGGZJNSVWA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:07.995196515 (628h13m7.86627956s) [noise=true , address=145.79.227.2:4114 , piece_id=A2AMMEDUDQGL7FW47MYKS25YHSB3M4EAZUJU7XA67QS3HEUCC45A, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.004321610 (628h13m7.875404655s) [noise=true , address=145.79.227.50:4374 , piece_id=FQ5YIJOGQASFPFA6M7UPCLL4TKUEDB6WUTP3G7GZRRVMUQXBL3VA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.017080365 (628h13m7.8881634s) [noise=true , address=145.79.227.24:4583 , piece_id=AESUXVKBYKDYDKJG7EK76UKRTYT44N2CZWEMGSU6HDLS5UAHBWEQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.027737497 (628h13m7.898820542s) [noise=true , address=145.79.227.15:4876 , piece_id=EP55WINK3OOLCR2AW7HA4XSCKAY3NT7TR2QQ5QU4Y3XIN6OXU6FA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.032006186 (628h13m7.903089221s) [noise=true , address=145.79.227.32:4835 , piece_id=4R6B2WBQWI5P6TXONXOFUQFSJZODOOKQLOE4ANQ5FZX7J5KP3QXA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.064492163 (628h13m7.935575209s) [noise=true , address=145.79.227.10:4401 , piece_id=ZRR7YLF2YZGYN4GON7HRWHHV23OYI5WYA7TUYZTWVZUKJWRHETNQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.157455087 (628h13m8.028538132s) [noise=true , address=77.90.31.21:4877 , piece_id=DSPYWJ7BQ7JYALHQLSMKYG2WZJHRFCTIQAAFDH57SLDN6LIX4BCQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.170049649 (628h13m8.041132694s) [noise=true , address=77.90.31.16:4735 , piece_id=NL4PZIASTKCFBF3J3CMVAFUOJDRAENWKMZ5H4TGJZ6MZ5S472X6Q, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.238002289 (628h13m8.109085334s) [noise=true , address=77.90.31.19:4841 , piece_id=WPK5YBWYJWKP6GDL4DXMWSCQGT5DHOKPQIHLQQIEK2GBRBYYPMQA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.351144357 (628h13m8.222227402s) [noise=true , address=45.91.116.7:4010 , piece_id=AYPDZW73FOAIIGRDWRZ562KW3TB33ZS5OC6UWV5RZ35T5WQBORFA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: <nil> Dec 31 19:42:08.351165226 (628h13m8.222248261s) [seg_pos={0 0}, upload=550486]: Segment reached optimal threshold of 13 pieces. Dec 31 19:42:08.351359487 (628h13m8.222442522s) [noise=true , address=45.91.116.5:4298 , piece_id=WECEEDNWTD7WDJQMPE3BH6X4QPT474E2XG4SOYYYMQ5DY7XMY3VQ, seg_pos={0 0}, upload=550486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 31 19:42:08.351365708 (628h13m8.222448743s) [noise=true , address=45.91.116.5:4298 , piece_id=WECEEDNWTD7WDJQMPE3BH6X4QPT474E2XG4SOYYYMQ5DY7XMY3VQ, seg_pos={0 0}, upload=550486]: Long tail cancellation due to optimal threshold Dec 31 19:42:08.351587441 (628h13m8.222670486s) [noise=true , address=45.91.116.10:4083 , piece_id=OTIBS53ZZKEYSLZWVDM6VT6D7LLVTVRFIBCQCPQ2IIV6IJWYSEPA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 19:42:08.351597069 (628h13m8.222680114s) [noise=true , address=45.91.116.10:4083 , piece_id=OTIBS53ZZKEYSLZWVDM6VT6D7LLVTVRFIBCQCPQ2IIV6IJWYSEPA, seg_pos={0 0}, upload=550486]: Long tail cancellation due to optimal threshold Dec 31 19:42:08.351635182 (628h13m8.222718217s) [noise=true , address=45.91.116.4:4940 , piece_id=DXP7SAMSFQQK7ZY6K5X7ADMBXGGY5UGTEODB4CE2ZQDZEMJRQPUA, seg_pos={0 0}, upload=550486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 19:42:08.351656401 (628h13m8.222739447s) [noise=true , address=45.91.116.4:4940 , piece_id=DXP7SAMSFQQK7ZY6K5X7ADMBXGGY5UGTEODB4CE2ZQDZEMJRQPUA, seg_pos={0 0}, upload=550486]: Long tail cancellation due to optimal threshold Dec 31 19:42:08.351691649 (628h13m8.222774694s) [seg_pos={0 0}, upload=550486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:42:08.351804134 (628h13m8.222887179s) [upload=550486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:42:08.351811187 (628h13m8.222894232s) [upload=550486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:42:08.645706803 (628h13m8.516789848s) [upload=550486]: Done uploading segments... Dec 31 19:42:08.645730157 (628h13m8.516813202s) [upload=550486]: Upload finished. err: <nil> Duration: 30.002602225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X........X.....................] duration: 31.651306529s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 31.651356896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...XX..........XX.X..X.XX....XXX...XX.X.......XXXXXX.......] duration: 21.165366684s trace: https://jaeger.dev.storj.io/trace/fe97b1872fb382e FAIL (2025-12-31 19:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: W6ZCXKWGB6TRX4RZIEU2BSHKBNLG47J4J5UYBP27VT4GSYXW5BXA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: C5T6UFF5AJM5YR7ZE7PMMJWWW5AMYY5GOHUFL77PEUUKLZCS4SFQ) context deadline exceeded ======LOGS====== Dec 31 19:36:52.249383499 (628h7m52.120466534s) [upload=550405]: Starting upload Dec 31 19:36:52.249451298 (628h7m52.120534333s) [upload=550405]: Done starting upload Dec 31 19:36:52.249478139 (628h7m52.120561174s) [upload=550405]: Uploading segments... Dec 31 19:36:52.249646529 (628h7m52.120729575s) [upload=550405]: Got next segment. Inline: false Dec 31 19:36:52.249688831 (628h7m52.120771866s) [upload=550405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:36:52.249765336 (628h7m52.120848391s) [upload=550405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:36:52.496545639 (628h7m52.367628684s) [seg_pos={0 0}, upload=550405]: Begin upload segment... Dec 31 19:36:52.496613768 (628h7m52.367696803s) [seg_pos={0 0}, upload=550405]: Done begin upload segment. Dec 31 19:36:52.496679524 (628h7m52.367762569s) [noise=true , address=45.91.116.14:4192 , piece_id=SSZZWYMOBTOM66ZQRW2V2QHU5LTSNKYGI3SP5R6DXS7ZFFKSOH3Q, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.496840290 (628h7m52.367923325s) [noise=true , address=45.91.116.6:4974 , piece_id=C5T6UFF5AJM5YR7ZE7PMMJWWW5AMYY5GOHUFL77PEUUKLZCS4SFQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.496925913 (628h7m52.368008948s) [noise=true , address=145.79.227.19:4078 , piece_id=BBBGMNJ65ZGK3JG6QB3UYN6HUZFW6CCOJ2L4BVFG6CCOJHUFBG6Q, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.497378164 (628h7m52.368461199s) [noise=true , address=145.79.227.5:4223 , piece_id=N32AXKSCZCNGZKUV2VIRSGJ6PCT5GD3V6L7AWJV6KMG6IINGZAJQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.497880541 (628h7m52.368963586s) [noise=true , address=145.79.227.48:4301 , piece_id=5MQ44QK3HZBJ4O27MGN7NLC6SW3XVTOMNQFFSPTKCVWAGSZ5VJPQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.498439265 (628h7m52.36952231s) [noise=true , address=77.90.31.21:4877 , piece_id=HEHBMMDKUVIWJ4ODSGYGKWM5FRGE2R4RGKRUZHWL4WOYGOOYASSQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.498877649 (628h7m52.369960694s) [noise=true , address=45.91.116.9:4263 , piece_id=HMW3UA2YCWXM52EKNRJCVBDKX6V4RPEOQXJJ6JAYV6UHJZER3GHA, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.499264597 (628h7m52.370347642s) [noise=true , address=145.79.227.17:4971 , piece_id=BFTXHCVFDCDRPZRC6SJ76OOG3F2ABCEMYQFTLDJPVO7USZN2I4IA, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.499701087 (628h7m52.370784122s) [noise=true , address=145.79.227.52:4445 , piece_id=CDHGS6WMNK65GZJETIUOV4SSRJAIIRUUQK3FPG3NQUVQQNKOXA4A, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.502341846 (628h7m52.373424881s) [noise=true , address=145.79.227.51:4410 , piece_id=KB2JLDYPRL2DT3IPXTPGH2LUVYYGU3IOW5M54F3ZJ4KUVQIJN7OQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.503739437 (628h7m52.374822472s) [noise=true , address=145.79.227.11:4729 , piece_id=S3GGD2CMT4GJZW3EPGVCXSXRMJRPZUV2RPD5YZUB37MODXWJ2PNA, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.505023061 (628h7m52.376106096s) [noise=true , address=145.79.227.50:4375 , piece_id=K3XSBGQ4BRT2KDCQ6MX6GB5K6FPG4R3S7FY6WRNSP72EEENX766Q, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.506510364 (628h7m52.377593399s) [noise=true , address=145.79.227.36:4518 , piece_id=EQSQIKKDWBQDPZN7I7DYJ7GCVUYUBA3B3TLZR6TN7VF3J22Y2GMQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.507023911 (628h7m52.378106946s) [noise=true , address=145.79.227.41:4085 , piece_id=AJLHMXOUWSPMV3Q4KFB2LDER42DTGNP3POMJ3QDEPI7NRZM2M7EQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.507241505 (628h7m52.37832454s) [noise=true , address=77.90.31.16:4735 , piece_id=W6ZCXKWGB6TRX4RZIEU2BSHKBNLG47J4J5UYBP27VT4GSYXW5BXA, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:52.508957383 (628h7m52.380040418s) [noise=true , address=145.79.227.13:4803 , piece_id=JDC2XMJEWJOAMLWQA6D6HQRATISEGJLDSPHUJSN2LYWGJ2HYNPHQ, seg_pos={0 0}, upload=550405]: Uploading piece... Dec 31 19:36:54.206102252 (628h7m54.077185298s) [upload=550405]: Next returned nil segment Dec 31 19:36:54.206122731 (628h7m54.077205766s) [upload=550405]: Waiting for error group managing segments... Dec 31 19:36:54.488957306 (628h7m54.360040371s) [noise=true , address=145.79.227.17:4971 , piece_id=BFTXHCVFDCDRPZRC6SJ76OOG3F2ABCEMYQFTLDJPVO7USZN2I4IA, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.492006652 (628h7m54.363089687s) [noise=true , address=145.79.227.5:4223 , piece_id=N32AXKSCZCNGZKUV2VIRSGJ6PCT5GD3V6L7AWJV6KMG6IINGZAJQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.503872867 (628h7m54.374955902s) [noise=true , address=145.79.227.51:4410 , piece_id=KB2JLDYPRL2DT3IPXTPGH2LUVYYGU3IOW5M54F3ZJ4KUVQIJN7OQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.506067606 (628h7m54.377150641s) [noise=true , address=145.79.227.13:4803 , piece_id=JDC2XMJEWJOAMLWQA6D6HQRATISEGJLDSPHUJSN2LYWGJ2HYNPHQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.516406302 (628h7m54.387489347s) [noise=true , address=145.79.227.50:4375 , piece_id=K3XSBGQ4BRT2KDCQ6MX6GB5K6FPG4R3S7FY6WRNSP72EEENX766Q, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.522420033 (628h7m54.393503078s) [noise=true , address=145.79.227.36:4518 , piece_id=EQSQIKKDWBQDPZN7I7DYJ7GCVUYUBA3B3TLZR6TN7VF3J22Y2GMQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.528849145 (628h7m54.39993218s) [noise=true , address=145.79.227.19:4078 , piece_id=BBBGMNJ65ZGK3JG6QB3UYN6HUZFW6CCOJ2L4BVFG6CCOJHUFBG6Q, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.554591172 (628h7m54.425674217s) [noise=true , address=145.79.227.11:4729 , piece_id=S3GGD2CMT4GJZW3EPGVCXSXRMJRPZUV2RPD5YZUB37MODXWJ2PNA, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.570433267 (628h7m54.441516302s) [noise=true , address=145.79.227.48:4301 , piece_id=5MQ44QK3HZBJ4O27MGN7NLC6SW3XVTOMNQFFSPTKCVWAGSZ5VJPQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.581780684 (628h7m54.452863729s) [noise=true , address=145.79.227.41:4085 , piece_id=AJLHMXOUWSPMV3Q4KFB2LDER42DTGNP3POMJ3QDEPI7NRZM2M7EQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.589125288 (628h7m54.460208334s) [noise=true , address=145.79.227.52:4445 , piece_id=CDHGS6WMNK65GZJETIUOV4SSRJAIIRUUQK3FPG3NQUVQQNKOXA4A, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.682890600 (628h7m54.553973635s) [noise=true , address=77.90.31.16:4735 , piece_id=W6ZCXKWGB6TRX4RZIEU2BSHKBNLG47J4J5UYBP27VT4GSYXW5BXA, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.800422848 (628h7m54.671505883s) [noise=true , address=45.91.116.6:4974 , piece_id=C5T6UFF5AJM5YR7ZE7PMMJWWW5AMYY5GOHUFL77PEUUKLZCS4SFQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: <nil> Dec 31 19:36:54.800441323 (628h7m54.671524358s) [seg_pos={0 0}, upload=550405]: Segment reached optimal threshold of 13 pieces. Dec 31 19:36:54.800660460 (628h7m54.671743495s) [noise=true , address=45.91.116.14:4192 , piece_id=SSZZWYMOBTOM66ZQRW2V2QHU5LTSNKYGI3SP5R6DXS7ZFFKSOH3Q, seg_pos={0 0}, upload=550405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 31 19:36:54.800667824 (628h7m54.671750869s) [noise=true , address=45.91.116.14:4192 , piece_id=SSZZWYMOBTOM66ZQRW2V2QHU5LTSNKYGI3SP5R6DXS7ZFFKSOH3Q, seg_pos={0 0}, upload=550405]: Long tail cancellation due to optimal threshold Dec 31 19:36:54.800893303 (628h7m54.671976338s) [noise=true , address=45.91.116.9:4263 , piece_id=HMW3UA2YCWXM52EKNRJCVBDKX6V4RPEOQXJJ6JAYV6UHJZER3GHA, seg_pos={0 0}, upload=550405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:36:54.800901008 (628h7m54.671984043s) [noise=true , address=45.91.116.9:4263 , piece_id=HMW3UA2YCWXM52EKNRJCVBDKX6V4RPEOQXJJ6JAYV6UHJZER3GHA, seg_pos={0 0}, upload=550405]: Long tail cancellation due to optimal threshold Dec 31 19:36:54.800956453 (628h7m54.672039488s) [noise=true , address=77.90.31.21:4877 , piece_id=HEHBMMDKUVIWJ4ODSGYGKWM5FRGE2R4RGKRUZHWL4WOYGOOYASSQ, seg_pos={0 0}, upload=550405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:48294->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:36:54.800965992 (628h7m54.672049027s) [noise=true , address=77.90.31.21:4877 , piece_id=HEHBMMDKUVIWJ4ODSGYGKWM5FRGE2R4RGKRUZHWL4WOYGOOYASSQ, seg_pos={0 0}, upload=550405]: Long tail cancellation due to optimal threshold Dec 31 19:36:54.800984737 (628h7m54.672067782s) [seg_pos={0 0}, upload=550405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:36:54.801096420 (628h7m54.672179455s) [upload=550405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:36:54.801116388 (628h7m54.672199433s) [upload=550405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:36:55.089337489 (628h7m54.960420534s) [upload=550405]: Done uploading segments... Dec 31 19:36:55.089355975 (628h7m54.96043901s) [upload=550405]: Upload finished. err: <nil> Duration: 21.165389347s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)