Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
49.436656116s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:23:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 49.436681824s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX] duration:
5.001029839s trace:
https://jaeger.dev.storj.io/trace/30d62970604527ba
FAIL (2025-12-29 16:23:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001058033s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX....XX.......X....X.X......X.....X......X...XX.XXX....X.] duration:
6.025229792s trace:
https://jaeger.dev.storj.io/trace/2de2594c4d6e394a
FAIL (2025-12-29 16:23:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:22:14.536640538 (576h53m14.407723573s) [upload=505721]: Starting upload
Dec 29 16:22:14.536718607 (576h53m14.407801673s) [upload=505721]: Done starting upload
Dec 29 16:22:14.536834017 (576h53m14.407917062s) [upload=505721]: Uploading segments...
Dec 29 16:22:14.536959626 (576h53m14.408042661s) [upload=505721]: Got next segment. Inline: false
Dec 29 16:22:14.537055619 (576h53m14.408138654s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:22:14.537062512 (576h53m14.408145557s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:22:14.786849233 (576h53m14.657932288s) [seg_pos={0 0}, upload=505721]: Begin upload segment...
Dec 29 16:22:14.786908106 (576h53m14.657991151s) [seg_pos={0 0}, upload=505721]: Done begin upload segment.
Dec 29 16:22:14.786941099 (576h53m14.658024144s) [noise=true , address=45.91.116.11:4118 , piece_id=NGOHIVMXVBWTLARDFBZP7BS3UFHQRKPRTQRY7DIZDFBISVABAJKQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.790304162 (576h53m14.661387217s) [noise=true , address=145.79.227.41:4086 , piece_id=RRYTE554JNYR3BSXLJLB63SKCIU7NDSVHYQO2I3NW5J3TFPKKZNA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.791339113 (576h53m14.662422158s) [noise=true , address=145.79.227.19:4079 , piece_id=AQUU2T5QP4U5ODRWK2KBMETZ4ZK5QOCKHOXC3WW6VEROERZHE56A, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.791904629 (576h53m14.662987664s) [noise=true , address=45.91.116.6:4973 , piece_id=4TWLX7YIHH5BGCY2BVFE47UWHHDJCFDFJF6DFWPZJTFKWODYRP5Q, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.792559947 (576h53m14.663642992s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.793272435 (576h53m14.66435548s) [noise=true , address=45.91.116.8:4045 , piece_id=7OF5VTQGBI6JKKYU5JZSC4HRWDOGB46LVDGFE5WLL3U5LQWBR64Q, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.793833583 (576h53m14.664916618s) [noise=true , address=77.90.31.19:4842 , piece_id=VPFE4566O6O7WTRFVHKNXSOZRNPIQY3FI2JT7E6ROQBSINOJC3AQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.794888652 (576h53m14.665971687s) [noise=true , address=145.79.227.2:4113 , piece_id=SOYIACZBSSJWSNFY76TKG22P7QUGBXQVCA7CQRQM56ZWZMNKVXLA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.794900123 (576h53m14.665983158s) [noise=true , address=145.79.227.13:4804 , piece_id=J7DPJ3CGRYNEQTH5HHAZCUU7RUKPNN4GXL2EAWXGPLCDTFW6EYHQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.795995429 (576h53m14.667078475s) [noise=true , address=145.79.227.4:4185 , piece_id=UMIDKC6T32QPYBMDDLQBZ4LRFDGOIOUX55JWDUQNLVTS5NIUPRVQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.797173162 (576h53m14.668256197s) [noise=true , address=45.91.116.9:4261 , piece_id=FDHJNTB42PXVFWGHBSBODZV4CSOKSX2CZJP6QE5T6XMCO6JFLNDQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.798455835 (576h53m14.66953888s) [noise=true , address=145.79.227.26:4653 , piece_id=NKS6FTYRQXIJ7DW74LD5OBOCO4MDLJVLBDAZ4T2UH6WHVWFCEB3A, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.799037582 (576h53m14.670120617s) [noise=true , address=145.79.227.52:4446 , piece_id=AGTX575OKIVSNC37L4RAMDN27X6EMQEVC32GNFFTCJMECPYPXFYQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.801439275 (576h53m14.67252232s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.808537852 (576h53m14.679620887s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.810049270 (576h53m14.681132315s) [noise=true , address=145.79.227.16:4910 , piece_id=GSZ2ZG4M22NF3GOIVH2MNRMJTY45ZY36FDCAKA5PU2VGPLUDJ2MA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:16.487519006 (576h53m16.358602041s) [upload=505721]: Next returned nil segment
Dec 29 16:22:16.487536800 (576h53m16.358619835s) [upload=505721]: Waiting for error group managing segments...
Dec 29 16:22:16.782156089 (576h53m16.653239124s) [noise=true , address=145.79.227.26:4653 , piece_id=NKS6FTYRQXIJ7DW74LD5OBOCO4MDLJVLBDAZ4T2UH6WHVWFCEB3A, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.784813189 (576h53m16.655896234s) [noise=true , address=145.79.227.19:4079 , piece_id=AQUU2T5QP4U5ODRWK2KBMETZ4ZK5QOCKHOXC3WW6VEROERZHE56A, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.787034529 (576h53m16.658117574s) [noise=true , address=145.79.227.41:4086 , piece_id=RRYTE554JNYR3BSXLJLB63SKCIU7NDSVHYQO2I3NW5J3TFPKKZNA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.787911258 (576h53m16.658994293s) [noise=true , address=145.79.227.2:4113 , piece_id=SOYIACZBSSJWSNFY76TKG22P7QUGBXQVCA7CQRQM56ZWZMNKVXLA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.796760800 (576h53m16.667843835s) [noise=true , address=145.79.227.4:4185 , piece_id=UMIDKC6T32QPYBMDDLQBZ4LRFDGOIOUX55JWDUQNLVTS5NIUPRVQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.797484828 (576h53m16.668567883s) [noise=true , address=145.79.227.13:4804 , piece_id=J7DPJ3CGRYNEQTH5HHAZCUU7RUKPNN4GXL2EAWXGPLCDTFW6EYHQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.812412232 (576h53m16.683495277s) [noise=true , address=145.79.227.52:4446 , piece_id=AGTX575OKIVSNC37L4RAMDN27X6EMQEVC32GNFFTCJMECPYPXFYQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.842032754 (576h53m16.713115789s) [noise=true , address=145.79.227.16:4910 , piece_id=GSZ2ZG4M22NF3GOIVH2MNRMJTY45ZY36FDCAKA5PU2VGPLUDJ2MA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.949728963 (576h53m16.820811998s) [noise=true , address=77.90.31.19:4842 , piece_id=VPFE4566O6O7WTRFVHKNXSOZRNPIQY3FI2JT7E6ROQBSINOJC3AQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.032032773 (576h53m16.903115808s) [noise=true , address=45.91.116.6:4973 , piece_id=4TWLX7YIHH5BGCY2BVFE47UWHHDJCFDFJF6DFWPZJTFKWODYRP5Q, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.040020553 (576h53m16.911103598s) [noise=true , address=45.91.116.9:4261 , piece_id=FDHJNTB42PXVFWGHBSBODZV4CSOKSX2CZJP6QE5T6XMCO6JFLNDQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.141475789 (576h53m17.012558824s) [noise=true , address=45.91.116.11:4118 , piece_id=NGOHIVMXVBWTLARDFBZP7BS3UFHQRKPRTQRY7DIZDFBISVABAJKQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.858246329 (576h53m17.729329374s) [noise=true , address=45.91.116.8:4045 , piece_id=7OF5VTQGBI6JKKYU5JZSC4HRWDOGB46LVDGFE5WLL3U5LQWBR64Q, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.858271577 (576h53m17.729354612s) [seg_pos={0 0}, upload=505721]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:22:17.858502657 (576h53m17.729585692s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858510983 (576h53m17.729594028s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858746020 (576h53m17.729829066s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858769836 (576h53m17.729852901s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858790535 (576h53m17.72987372s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858801205 (576h53m17.72988427s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858821023 (576h53m17.729904058s) [seg_pos={0 0}, upload=505721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:22:17.858944218 (576h53m17.730027263s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:22:17.858952183 (576h53m17.730035218s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:22:18.158597413 (576h53m18.029680448s) [upload=505721]: Done uploading segments...
Dec 29 16:22:18.158621889 (576h53m18.029704935s) [upload=505721]: Upload finished. err: <nil>
Duration: 6.025259618s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XX] duration:
5.001385415s trace:
https://jaeger.dev.storj.io/trace/585a95e86e6e3c46
FAIL (2025-12-29 16:22:57 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001408639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..XX....XX.......X....X.X......X.....X......X...XX.XXX....X] duration:
30.002684924s trace:
https://jaeger.dev.storj.io/trace/2de2594c4d6e394a
FAIL (2025-12-29 16:22:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:22:14.536640538 (576h53m14.407723573s) [upload=505721]: Starting upload
Dec 29 16:22:14.536718607 (576h53m14.407801673s) [upload=505721]: Done starting upload
Dec 29 16:22:14.536834017 (576h53m14.407917062s) [upload=505721]: Uploading segments...
Dec 29 16:22:14.536959626 (576h53m14.408042661s) [upload=505721]: Got next segment. Inline: false
Dec 29 16:22:14.537055619 (576h53m14.408138654s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:22:14.537062512 (576h53m14.408145557s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:22:14.786849233 (576h53m14.657932288s) [seg_pos={0 0}, upload=505721]: Begin upload segment...
Dec 29 16:22:14.786908106 (576h53m14.657991151s) [seg_pos={0 0}, upload=505721]: Done begin upload segment.
Dec 29 16:22:14.786941099 (576h53m14.658024144s) [noise=true , address=45.91.116.11:4118 , piece_id=NGOHIVMXVBWTLARDFBZP7BS3UFHQRKPRTQRY7DIZDFBISVABAJKQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.790304162 (576h53m14.661387217s) [noise=true , address=145.79.227.41:4086 , piece_id=RRYTE554JNYR3BSXLJLB63SKCIU7NDSVHYQO2I3NW5J3TFPKKZNA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.791339113 (576h53m14.662422158s) [noise=true , address=145.79.227.19:4079 , piece_id=AQUU2T5QP4U5ODRWK2KBMETZ4ZK5QOCKHOXC3WW6VEROERZHE56A, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.791904629 (576h53m14.662987664s) [noise=true , address=45.91.116.6:4973 , piece_id=4TWLX7YIHH5BGCY2BVFE47UWHHDJCFDFJF6DFWPZJTFKWODYRP5Q, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.792559947 (576h53m14.663642992s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.793272435 (576h53m14.66435548s) [noise=true , address=45.91.116.8:4045 , piece_id=7OF5VTQGBI6JKKYU5JZSC4HRWDOGB46LVDGFE5WLL3U5LQWBR64Q, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.793833583 (576h53m14.664916618s) [noise=true , address=77.90.31.19:4842 , piece_id=VPFE4566O6O7WTRFVHKNXSOZRNPIQY3FI2JT7E6ROQBSINOJC3AQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.794888652 (576h53m14.665971687s) [noise=true , address=145.79.227.2:4113 , piece_id=SOYIACZBSSJWSNFY76TKG22P7QUGBXQVCA7CQRQM56ZWZMNKVXLA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.794900123 (576h53m14.665983158s) [noise=true , address=145.79.227.13:4804 , piece_id=J7DPJ3CGRYNEQTH5HHAZCUU7RUKPNN4GXL2EAWXGPLCDTFW6EYHQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.795995429 (576h53m14.667078475s) [noise=true , address=145.79.227.4:4185 , piece_id=UMIDKC6T32QPYBMDDLQBZ4LRFDGOIOUX55JWDUQNLVTS5NIUPRVQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.797173162 (576h53m14.668256197s) [noise=true , address=45.91.116.9:4261 , piece_id=FDHJNTB42PXVFWGHBSBODZV4CSOKSX2CZJP6QE5T6XMCO6JFLNDQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.798455835 (576h53m14.66953888s) [noise=true , address=145.79.227.26:4653 , piece_id=NKS6FTYRQXIJ7DW74LD5OBOCO4MDLJVLBDAZ4T2UH6WHVWFCEB3A, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.799037582 (576h53m14.670120617s) [noise=true , address=145.79.227.52:4446 , piece_id=AGTX575OKIVSNC37L4RAMDN27X6EMQEVC32GNFFTCJMECPYPXFYQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.801439275 (576h53m14.67252232s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.808537852 (576h53m14.679620887s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:14.810049270 (576h53m14.681132315s) [noise=true , address=145.79.227.16:4910 , piece_id=GSZ2ZG4M22NF3GOIVH2MNRMJTY45ZY36FDCAKA5PU2VGPLUDJ2MA, seg_pos={0 0}, upload=505721]: Uploading piece...
Dec 29 16:22:16.487519006 (576h53m16.358602041s) [upload=505721]: Next returned nil segment
Dec 29 16:22:16.487536800 (576h53m16.358619835s) [upload=505721]: Waiting for error group managing segments...
Dec 29 16:22:16.782156089 (576h53m16.653239124s) [noise=true , address=145.79.227.26:4653 , piece_id=NKS6FTYRQXIJ7DW74LD5OBOCO4MDLJVLBDAZ4T2UH6WHVWFCEB3A, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.784813189 (576h53m16.655896234s) [noise=true , address=145.79.227.19:4079 , piece_id=AQUU2T5QP4U5ODRWK2KBMETZ4ZK5QOCKHOXC3WW6VEROERZHE56A, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.787034529 (576h53m16.658117574s) [noise=true , address=145.79.227.41:4086 , piece_id=RRYTE554JNYR3BSXLJLB63SKCIU7NDSVHYQO2I3NW5J3TFPKKZNA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.787911258 (576h53m16.658994293s) [noise=true , address=145.79.227.2:4113 , piece_id=SOYIACZBSSJWSNFY76TKG22P7QUGBXQVCA7CQRQM56ZWZMNKVXLA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.796760800 (576h53m16.667843835s) [noise=true , address=145.79.227.4:4185 , piece_id=UMIDKC6T32QPYBMDDLQBZ4LRFDGOIOUX55JWDUQNLVTS5NIUPRVQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.797484828 (576h53m16.668567883s) [noise=true , address=145.79.227.13:4804 , piece_id=J7DPJ3CGRYNEQTH5HHAZCUU7RUKPNN4GXL2EAWXGPLCDTFW6EYHQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.812412232 (576h53m16.683495277s) [noise=true , address=145.79.227.52:4446 , piece_id=AGTX575OKIVSNC37L4RAMDN27X6EMQEVC32GNFFTCJMECPYPXFYQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.842032754 (576h53m16.713115789s) [noise=true , address=145.79.227.16:4910 , piece_id=GSZ2ZG4M22NF3GOIVH2MNRMJTY45ZY36FDCAKA5PU2VGPLUDJ2MA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:16.949728963 (576h53m16.820811998s) [noise=true , address=77.90.31.19:4842 , piece_id=VPFE4566O6O7WTRFVHKNXSOZRNPIQY3FI2JT7E6ROQBSINOJC3AQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.032032773 (576h53m16.903115808s) [noise=true , address=45.91.116.6:4973 , piece_id=4TWLX7YIHH5BGCY2BVFE47UWHHDJCFDFJF6DFWPZJTFKWODYRP5Q, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.040020553 (576h53m16.911103598s) [noise=true , address=45.91.116.9:4261 , piece_id=FDHJNTB42PXVFWGHBSBODZV4CSOKSX2CZJP6QE5T6XMCO6JFLNDQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.141475789 (576h53m17.012558824s) [noise=true , address=45.91.116.11:4118 , piece_id=NGOHIVMXVBWTLARDFBZP7BS3UFHQRKPRTQRY7DIZDFBISVABAJKQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.858246329 (576h53m17.729329374s) [noise=true , address=45.91.116.8:4045 , piece_id=7OF5VTQGBI6JKKYU5JZSC4HRWDOGB46LVDGFE5WLL3U5LQWBR64Q, seg_pos={0 0}, upload=505721]: Done uploading piece. err: <nil>
Dec 29 16:22:17.858271577 (576h53m17.729354612s) [seg_pos={0 0}, upload=505721]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:22:17.858502657 (576h53m17.729585692s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858510983 (576h53m17.729594028s) [noise=true , address=145.79.227.40:4049 , piece_id=GO4PDOT7YIY4OB7IDKJV5UJQ3BCTHN7Z33PFDPFKW7HO37A4U5EA, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858746020 (576h53m17.729829066s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858769836 (576h53m17.729852901s) [noise=true , address=145.79.227.44:4193 , piece_id=TTGXD6A3UXQYFFK4HCZD6OPLUOXJG53GT7DYUPK5ALGG3PSEAFKQ, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858790535 (576h53m17.72987372s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF
Dec 29 16:22:17.858801205 (576h53m17.72988427s) [noise=true , address=145.79.227.27:4690 , piece_id=DGCW4POMWBHDMQQ7DG3VNR4RZZ6AGRNMVKSJEOLNHVGY2IACAPLQ, seg_pos={0 0}, upload=505721]: Long tail cancellation due to optimal threshold
Dec 29 16:22:17.858821023 (576h53m17.729904058s) [seg_pos={0 0}, upload=505721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:22:17.858944218 (576h53m17.730027263s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:22:17.858952183 (576h53m17.730035218s) [upload=505721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:22:18.158597413 (576h53m18.029680448s) [upload=505721]: Done uploading segments...
Dec 29 16:22:18.158621889 (576h53m18.029704935s) [upload=505721]: Upload finished. err: <nil>
Duration: 30.002711264s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
40.808617076s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:22:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 40.808646182s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.X] duration:
5.000721953s trace:
https://jaeger.dev.storj.io/trace/6c0b530a0c628f13
FAIL (2025-12-29 16:22:05 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000743303s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X..XX....XX.......X....X.X......X.....X......X...XX.XXX....] duration:
5.755383623s trace:
https://jaeger.dev.storj.io/trace/10b8b35dca803716
FAIL (2025-12-29 16:21:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A) context deadline exceeded
======LOGS======
16:18:34.604001494 (576h49m34.47508454s) [upload=505672]: Starting upload
Dec 29 16:18:34.604084744 (576h49m34.475167779s) [upload=505672]: Done starting upload
Dec 29 16:18:34.604217736 (576h49m34.475300801s) [upload=505672]: Uploading segments...
Dec 29 16:18:34.604326724 (576h49m34.475409759s) [upload=505672]: Got next segment. Inline: false
Dec 29 16:18:34.604381779 (576h49m34.475464824s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:18:34.604387269 (576h49m34.475470314s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:18:34.854726944 (576h49m34.72580998s) [seg_pos={0 0}, upload=505672]: Begin upload segment...
Dec 29 16:18:34.854775697 (576h49m34.725858732s) [seg_pos={0 0}, upload=505672]: Done begin upload segment.
Dec 29 16:18:34.854804742 (576h49m34.725887787s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.854813529 (576h49m34.725896564s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855393063 (576h49m34.726476108s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855828822 (576h49m34.726911867s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.856226228 (576h49m34.727309263s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857106655 (576h49m34.728189691s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857257973 (576h49m34.728341008s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857703672 (576h49m34.728786717s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857921527 (576h49m34.729004572s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.858861106 (576h49m34.729944151s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.859838046 (576h49m34.730921091s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.860035883 (576h49m34.731118928s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862023598 (576h49m34.733106633s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862595136 (576h49m34.733678181s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.863884942 (576h49m34.734967987s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.864597439 (576h49m34.735680484s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:36.557194751 (576h49m36.428277786s) [upload=505672]: Next returned nil segment
Dec 29 16:18:36.557215791 (576h49m36.428298826s) [upload=505672]: Waiting for error group managing segments...
Dec 29 16:18:36.840378455 (576h49m36.711461501s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845225816 (576h49m36.716308851s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845492544 (576h49m36.716575589s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.856381368 (576h49m36.727464403s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.865134986 (576h49m36.736218021s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.869410227 (576h49m36.740493262s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890073363 (576h49m36.761156398s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890903423 (576h49m36.761986458s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.125466657 (576h49m36.996549692s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.142348191 (576h49m37.013431226s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.154086083 (576h49m37.025169118s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.465914331 (576h49m37.336997366s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789093502 (576h49m37.660176537s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789113521 (576h49m37.660196556s) [seg_pos={0 0}, upload=505672]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:18:37.789285448 (576h49m37.660368483s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection
Dec 29 16:18:37.789292521 (576h49m37.660375557s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789402140 (576h49m37.660485185s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789412349 (576h49m37.660495404s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789441825 (576h49m37.660524871s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789449560 (576h49m37.660532615s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789536827 (576h49m37.660619862s) [seg_pos={0 0}, upload=505672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:18:37.789624894 (576h49m37.660707929s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:18:37.789630575 (576h49m37.660713611s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:18:38.089988613 (576h49m37.961071648s) [upload=505672]: Done uploading segments...
Dec 29 16:18:38.090008241 (576h49m37.961091276s) [upload=505672]: Upload finished. err: <nil>
Duration: 5.755411755s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
34.31856048s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:21:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 34.318586901s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.X..XX....XX.......X....X.X......X.....X......X...XX.XXX...] duration:
5.758690376s trace:
https://jaeger.dev.storj.io/trace/10b8b35dca803716
FAIL (2025-12-29 16:21:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A) context deadline exceeded
======LOGS======
16:18:34.604001494 (576h49m34.47508454s) [upload=505672]: Starting upload
Dec 29 16:18:34.604084744 (576h49m34.475167779s) [upload=505672]: Done starting upload
Dec 29 16:18:34.604217736 (576h49m34.475300801s) [upload=505672]: Uploading segments...
Dec 29 16:18:34.604326724 (576h49m34.475409759s) [upload=505672]: Got next segment. Inline: false
Dec 29 16:18:34.604381779 (576h49m34.475464824s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:18:34.604387269 (576h49m34.475470314s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:18:34.854726944 (576h49m34.72580998s) [seg_pos={0 0}, upload=505672]: Begin upload segment...
Dec 29 16:18:34.854775697 (576h49m34.725858732s) [seg_pos={0 0}, upload=505672]: Done begin upload segment.
Dec 29 16:18:34.854804742 (576h49m34.725887787s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.854813529 (576h49m34.725896564s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855393063 (576h49m34.726476108s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855828822 (576h49m34.726911867s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.856226228 (576h49m34.727309263s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857106655 (576h49m34.728189691s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857257973 (576h49m34.728341008s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857703672 (576h49m34.728786717s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857921527 (576h49m34.729004572s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.858861106 (576h49m34.729944151s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.859838046 (576h49m34.730921091s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.860035883 (576h49m34.731118928s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862023598 (576h49m34.733106633s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862595136 (576h49m34.733678181s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.863884942 (576h49m34.734967987s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.864597439 (576h49m34.735680484s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:36.557194751 (576h49m36.428277786s) [upload=505672]: Next returned nil segment
Dec 29 16:18:36.557215791 (576h49m36.428298826s) [upload=505672]: Waiting for error group managing segments...
Dec 29 16:18:36.840378455 (576h49m36.711461501s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845225816 (576h49m36.716308851s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845492544 (576h49m36.716575589s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.856381368 (576h49m36.727464403s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.865134986 (576h49m36.736218021s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.869410227 (576h49m36.740493262s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890073363 (576h49m36.761156398s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890903423 (576h49m36.761986458s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.125466657 (576h49m36.996549692s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.142348191 (576h49m37.013431226s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.154086083 (576h49m37.025169118s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.465914331 (576h49m37.336997366s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789093502 (576h49m37.660176537s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789113521 (576h49m37.660196556s) [seg_pos={0 0}, upload=505672]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:18:37.789285448 (576h49m37.660368483s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection
Dec 29 16:18:37.789292521 (576h49m37.660375557s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789402140 (576h49m37.660485185s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789412349 (576h49m37.660495404s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789441825 (576h49m37.660524871s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789449560 (576h49m37.660532615s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789536827 (576h49m37.660619862s) [seg_pos={0 0}, upload=505672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:18:37.789624894 (576h49m37.660707929s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:18:37.789630575 (576h49m37.660713611s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:18:38.089988613 (576h49m37.961071648s) [upload=505672]: Done uploading segments...
Dec 29 16:18:38.090008241 (576h49m37.961091276s) [upload=505672]: Upload finished. err: <nil>
Duration: 5.758715764s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.] duration:
4.064929893s trace:
https://jaeger.dev.storj.io/trace/46b18f41e5630358
FAIL (2025-12-29 16:21:17 UTC)
orbiter warning: context deadline exceeded
Duration: 4.064954019s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X.X..XX....XX.......X....X.X......X.....X......X...XX.XXX..] duration:
5.709474698s trace:
https://jaeger.dev.storj.io/trace/10b8b35dca803716
FAIL (2025-12-29 16:21:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A) context deadline exceeded
======LOGS======
16:18:34.604001494 (576h49m34.47508454s) [upload=505672]: Starting upload
Dec 29 16:18:34.604084744 (576h49m34.475167779s) [upload=505672]: Done starting upload
Dec 29 16:18:34.604217736 (576h49m34.475300801s) [upload=505672]: Uploading segments...
Dec 29 16:18:34.604326724 (576h49m34.475409759s) [upload=505672]: Got next segment. Inline: false
Dec 29 16:18:34.604381779 (576h49m34.475464824s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:18:34.604387269 (576h49m34.475470314s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:18:34.854726944 (576h49m34.72580998s) [seg_pos={0 0}, upload=505672]: Begin upload segment...
Dec 29 16:18:34.854775697 (576h49m34.725858732s) [seg_pos={0 0}, upload=505672]: Done begin upload segment.
Dec 29 16:18:34.854804742 (576h49m34.725887787s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.854813529 (576h49m34.725896564s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855393063 (576h49m34.726476108s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855828822 (576h49m34.726911867s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.856226228 (576h49m34.727309263s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857106655 (576h49m34.728189691s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857257973 (576h49m34.728341008s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857703672 (576h49m34.728786717s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857921527 (576h49m34.729004572s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.858861106 (576h49m34.729944151s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.859838046 (576h49m34.730921091s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.860035883 (576h49m34.731118928s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862023598 (576h49m34.733106633s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862595136 (576h49m34.733678181s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.863884942 (576h49m34.734967987s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.864597439 (576h49m34.735680484s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:36.557194751 (576h49m36.428277786s) [upload=505672]: Next returned nil segment
Dec 29 16:18:36.557215791 (576h49m36.428298826s) [upload=505672]: Waiting for error group managing segments...
Dec 29 16:18:36.840378455 (576h49m36.711461501s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845225816 (576h49m36.716308851s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845492544 (576h49m36.716575589s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.856381368 (576h49m36.727464403s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.865134986 (576h49m36.736218021s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.869410227 (576h49m36.740493262s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890073363 (576h49m36.761156398s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890903423 (576h49m36.761986458s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.125466657 (576h49m36.996549692s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.142348191 (576h49m37.013431226s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.154086083 (576h49m37.025169118s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.465914331 (576h49m37.336997366s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789093502 (576h49m37.660176537s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789113521 (576h49m37.660196556s) [seg_pos={0 0}, upload=505672]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:18:37.789285448 (576h49m37.660368483s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection
Dec 29 16:18:37.789292521 (576h49m37.660375557s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789402140 (576h49m37.660485185s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789412349 (576h49m37.660495404s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789441825 (576h49m37.660524871s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789449560 (576h49m37.660532615s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789536827 (576h49m37.660619862s) [seg_pos={0 0}, upload=505672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:18:37.789624894 (576h49m37.660707929s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:18:37.789630575 (576h49m37.660713611s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:18:38.089988613 (576h49m37.961071648s) [upload=505672]: Done uploading segments...
Dec 29 16:18:38.090008241 (576h49m37.961091276s) [upload=505672]: Upload finished. err: <nil>
Duration: 5.709503282s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
50.358884992s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:20:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 50.358929056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX] duration:
5.001493323s trace:
https://jaeger.dev.storj.io/trace/46b18f41e5630358
FAIL (2025-12-29 16:20:15 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001520175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.X..XX....XX.......X....X.X......X.....X......X...XX.XXX.] duration:
5.870609716s trace:
https://jaeger.dev.storj.io/trace/10b8b35dca803716
FAIL (2025-12-29 16:19:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A) context deadline exceeded
======LOGS======
16:18:34.604001494 (576h49m34.47508454s) [upload=505672]: Starting upload
Dec 29 16:18:34.604084744 (576h49m34.475167779s) [upload=505672]: Done starting upload
Dec 29 16:18:34.604217736 (576h49m34.475300801s) [upload=505672]: Uploading segments...
Dec 29 16:18:34.604326724 (576h49m34.475409759s) [upload=505672]: Got next segment. Inline: false
Dec 29 16:18:34.604381779 (576h49m34.475464824s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:18:34.604387269 (576h49m34.475470314s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:18:34.854726944 (576h49m34.72580998s) [seg_pos={0 0}, upload=505672]: Begin upload segment...
Dec 29 16:18:34.854775697 (576h49m34.725858732s) [seg_pos={0 0}, upload=505672]: Done begin upload segment.
Dec 29 16:18:34.854804742 (576h49m34.725887787s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.854813529 (576h49m34.725896564s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855393063 (576h49m34.726476108s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855828822 (576h49m34.726911867s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.856226228 (576h49m34.727309263s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857106655 (576h49m34.728189691s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857257973 (576h49m34.728341008s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857703672 (576h49m34.728786717s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857921527 (576h49m34.729004572s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.858861106 (576h49m34.729944151s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.859838046 (576h49m34.730921091s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.860035883 (576h49m34.731118928s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862023598 (576h49m34.733106633s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862595136 (576h49m34.733678181s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.863884942 (576h49m34.734967987s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.864597439 (576h49m34.735680484s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:36.557194751 (576h49m36.428277786s) [upload=505672]: Next returned nil segment
Dec 29 16:18:36.557215791 (576h49m36.428298826s) [upload=505672]: Waiting for error group managing segments...
Dec 29 16:18:36.840378455 (576h49m36.711461501s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845225816 (576h49m36.716308851s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845492544 (576h49m36.716575589s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.856381368 (576h49m36.727464403s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.865134986 (576h49m36.736218021s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.869410227 (576h49m36.740493262s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890073363 (576h49m36.761156398s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890903423 (576h49m36.761986458s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.125466657 (576h49m36.996549692s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.142348191 (576h49m37.013431226s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.154086083 (576h49m37.025169118s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.465914331 (576h49m37.336997366s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789093502 (576h49m37.660176537s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789113521 (576h49m37.660196556s) [seg_pos={0 0}, upload=505672]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:18:37.789285448 (576h49m37.660368483s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection
Dec 29 16:18:37.789292521 (576h49m37.660375557s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789402140 (576h49m37.660485185s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789412349 (576h49m37.660495404s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789441825 (576h49m37.660524871s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789449560 (576h49m37.660532615s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789536827 (576h49m37.660619862s) [seg_pos={0 0}, upload=505672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:18:37.789624894 (576h49m37.660707929s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:18:37.789630575 (576h49m37.660713611s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:18:38.089988613 (576h49m37.961071648s) [upload=505672]: Done uploading segments...
Dec 29 16:18:38.090008241 (576h49m37.961091276s) [upload=505672]: Upload finished. err: <nil>
Duration: 5.870637067s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
31.795593492s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:19:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 31.795606457s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXX] duration:
5.001452564s trace:
https://jaeger.dev.storj.io/trace/2b2bb0b332201626
FAIL (2025-12-29 16:19:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001473494s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...X.X..XX....XX.......X....X.X......X.....X......X...XX.XXX] duration:
30.004073051s trace:
https://jaeger.dev.storj.io/trace/10b8b35dca803716
FAIL (2025-12-29 16:19:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A) context deadline exceeded
======LOGS======
16:18:34.604001494 (576h49m34.47508454s) [upload=505672]: Starting upload
Dec 29 16:18:34.604084744 (576h49m34.475167779s) [upload=505672]: Done starting upload
Dec 29 16:18:34.604217736 (576h49m34.475300801s) [upload=505672]: Uploading segments...
Dec 29 16:18:34.604326724 (576h49m34.475409759s) [upload=505672]: Got next segment. Inline: false
Dec 29 16:18:34.604381779 (576h49m34.475464824s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:18:34.604387269 (576h49m34.475470314s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:18:34.854726944 (576h49m34.72580998s) [seg_pos={0 0}, upload=505672]: Begin upload segment...
Dec 29 16:18:34.854775697 (576h49m34.725858732s) [seg_pos={0 0}, upload=505672]: Done begin upload segment.
Dec 29 16:18:34.854804742 (576h49m34.725887787s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.854813529 (576h49m34.725896564s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855393063 (576h49m34.726476108s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.855828822 (576h49m34.726911867s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.856226228 (576h49m34.727309263s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857106655 (576h49m34.728189691s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857257973 (576h49m34.728341008s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857703672 (576h49m34.728786717s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.857921527 (576h49m34.729004572s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.858861106 (576h49m34.729944151s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.859838046 (576h49m34.730921091s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.860035883 (576h49m34.731118928s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862023598 (576h49m34.733106633s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.862595136 (576h49m34.733678181s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.863884942 (576h49m34.734967987s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:34.864597439 (576h49m34.735680484s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Uploading piece...
Dec 29 16:18:36.557194751 (576h49m36.428277786s) [upload=505672]: Next returned nil segment
Dec 29 16:18:36.557215791 (576h49m36.428298826s) [upload=505672]: Waiting for error group managing segments...
Dec 29 16:18:36.840378455 (576h49m36.711461501s) [noise=true , address=145.79.227.15:4875 , piece_id=QBGT6VWXBMMIRVBLUJOF5L6YVZSN5PZHQFTLKMTWDSBUETWAL4PA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845225816 (576h49m36.716308851s) [noise=true , address=145.79.227.50:4375 , piece_id=L52KRXS6S2BYAB4G6TP5KOUCD5J34C6UICHWAGU7A5EIFTHNXFUA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.845492544 (576h49m36.716575589s) [noise=true , address=145.79.227.37:4565 , piece_id=EGJQ4ETXGZK4ZIJ7ILP4VWQ7VSP3SI43Y3S4BVYOEGP2MKUVHTTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.856381368 (576h49m36.727464403s) [noise=true , address=145.79.227.14:4837 , piece_id=SYN45US6YBTW56GTKOWVWUB6WEGPGVXW5LZQKO5THVWXWEQ3LMQA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.865134986 (576h49m36.736218021s) [noise=true , address=145.79.227.7:4294 , piece_id=BP6F7MI46FHYZSSPMQDRWN5DZ2LVCF4ABQVDAEH35KQUGVGJ4AGQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.869410227 (576h49m36.740493262s) [noise=true , address=145.79.227.51:4410 , piece_id=FSVHQVZAFXBTWJLLPKFFWKZNVMY7TX367GNU3SO344TX4LRBZ5VA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890073363 (576h49m36.761156398s) [noise=true , address=145.79.227.11:4729 , piece_id=NDHXFDU5K25TUOBBQG3TF2F3XFGYBN6SQTO7SFLO3XTCZPLM74GA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:36.890903423 (576h49m36.761986458s) [noise=true , address=145.79.227.19:4079 , piece_id=Y7DVGQWSQXPULOQEY3NZ7Y4A5CQJTZF524RSS2PALRUGP6CKYDKQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.125466657 (576h49m36.996549692s) [noise=true , address=45.91.116.8:4046 , piece_id=WXUXZNY4BMNXBZ32T4BOTXV7ZECZL34ZBYGKIOGWWNWFVGUFI55A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.142348191 (576h49m37.013431226s) [noise=true , address=77.90.31.10:4518 , piece_id=5CBUIGDSTKBAWZMBKV2MZYMA6VLXKS3E4HLEHAFE7FGFLAB7UT6A, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.154086083 (576h49m37.025169118s) [noise=true , address=77.90.31.11:4553 , piece_id=IOIMQUSKS2HV2KB6BJDLGG55R6IXPFN243EX6HRUCMMGVIP5GWNA, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.465914331 (576h49m37.336997366s) [noise=true , address=77.90.31.13:4629 , piece_id=3DEAB7IV55IV2A3EYY2TTASV6RSPT7O2LPISCHFD5S4I5IFAFFMQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789093502 (576h49m37.660176537s) [noise=true , address=77.90.31.19:4844 , piece_id=OUPWCHVMXIAIXEI2IZJXA5SW4EDBTW6N2QSMJYYIJO6IL5PVQMTQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: <nil>
Dec 29 16:18:37.789113521 (576h49m37.660196556s) [seg_pos={0 0}, upload=505672]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:18:37.789285448 (576h49m37.660368483s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->45.91.116.9:4262: use of closed network connection
Dec 29 16:18:37.789292521 (576h49m37.660375557s) [noise=true , address=45.91.116.9:4262 , piece_id=ANI6DLYOFBHNJ27RMPOI3HQI7JWGAQYTQCIFMXW2ODRGQKHUT5UQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789402140 (576h49m37.660485185s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789412349 (576h49m37.660495404s) [noise=true , address=45.91.116.11:4118 , piece_id=YZJM35EKA3KLUOND7OJMZLDW5L4GE2CK3XSKHJYSOYTAREEIPNWQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789441825 (576h49m37.660524871s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 29 16:18:37.789449560 (576h49m37.660532615s) [noise=true , address=45.91.116.13:4156 , piece_id=FB7EB53MPS4XGRH236QLM2IGX6W7SJYQW2AAJ6OI5MUJALECZ6RQ, seg_pos={0 0}, upload=505672]: Long tail cancellation due to optimal threshold
Dec 29 16:18:37.789536827 (576h49m37.660619862s) [seg_pos={0 0}, upload=505672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:18:37.789624894 (576h49m37.660707929s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:18:37.789630575 (576h49m37.660713611s) [upload=505672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:18:38.089988613 (576h49m37.961071648s) [upload=505672]: Done uploading segments...
Dec 29 16:18:38.090008241 (576h49m37.961091276s) [upload=505672]: Upload finished. err: <nil>
Duration: 30.004096596s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
58.494540169s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:18:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 58.494564284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XX] duration:
5.000723033s trace:
https://jaeger.dev.storj.io/trace/660fe170da644605
FAIL (2025-12-29 16:18:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000747089s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X...X.X..XX....XX.......X....X.X......X.....X......X...XX.XX] duration:
30.005010568s trace:
https://jaeger.dev.storj.io/trace/3edf6964623d7031
FAIL (2025-12-29 16:17:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: HJIVFVLP4QMEQTW75OB3ORYSDEB2P3F7SWUY5ULNMKVCVRDQBIZA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: XZZ4YTHJL24M2PKOOMLX45MMOLBIFH44TRXQT7B4XUL773REC4TQ) context deadline exceeded
======LOGS======
g segments...
Dec 29 16:17:27.381144427 (576h48m27.252227462s) [upload=505659]: Got next segment. Inline: false
Dec 29 16:17:27.381190015 (576h48m27.25227306s) [upload=505659]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:17:27.381195545 (576h48m27.25227859s) [upload=505659]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:17:27.642531898 (576h48m27.513614953s) [seg_pos={0 0}, upload=505659]: Begin upload segment...
Dec 29 16:17:27.642602883 (576h48m27.513685929s) [seg_pos={0 0}, upload=505659]: Done begin upload segment.
Dec 29 16:17:27.642638141 (576h48m27.513721186s) [noise=true , address=145.79.227.7:4295 , piece_id=TYS6GDJMD22CNIUGWDWOC25PSXWYYSKXTPFWBTQ6PH2IX6A4SBDA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.642650194 (576h48m27.513733239s) [noise=true , address=77.90.31.19:4841 , piece_id=XZZ4YTHJL24M2PKOOMLX45MMOLBIFH44TRXQT7B4XUL773REC4TQ, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.642707483 (576h48m27.513790528s) [noise=true , address=77.90.31.21:4877 , piece_id=43UPVJH4H6EXAK2OQZRVXXRPFVYG2QUXBPQLP3KYUMODIP6SG56A, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.643270484 (576h48m27.51435353s) [noise=true , address=77.90.31.14:4661 , piece_id=MUAKYXL5ELANCW3ZJGQP4MRGTKUAD26QVG3ZUZHS7XYCDUBIHIBA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.643448704 (576h48m27.514531749s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGR3I7DKJWMTF6CTEWAHQA7T2TBGCQZNMPJNKW2UTD5VB6QAEOMA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.643902729 (576h48m27.514985774s) [noise=true , address=45.91.116.10:4082 , piece_id=UQU2IK2ICBSQ4OVLEK7WNEB3QD55FW7BMG6C2MRPHVVBIFCCZWJQ, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.644555822 (576h48m27.515638867s) [noise=true , address=145.79.227.6:4258 , piece_id=MFITWLD7RZGJATOTKXAY3V3XFXSCCKJPLP3GS3O57GPNF2D6Z3TA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.644829984 (576h48m27.515913019s) [noise=true , address=45.91.116.6:4974 , piece_id=YDACDM2XNQXG2XLQRAG64UTCS5V236PONTFGGWP2ZY6OKFV2P4GA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.645554825 (576h48m27.51663787s) [noise=true , address=145.79.227.48:4303 , piece_id=XUJDEUWVQQUUBZHNH3ZATIV7A5IJEDWEJCCCK7TKGYHPXHTMUJWQ, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.646678374 (576h48m27.517761409s) [noise=true , address=45.91.116.12:4228 , piece_id=RJW4BARGCY5EF6ASSHP4NGRLVFOBXK4KSG4VB46QOYIFOGNAZA4A, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.646872173 (576h48m27.517955228s) [noise=true , address=145.79.227.30:4762 , piece_id=L4U7YR5LLSODCYAAEGO2C6GZV674R4JK6NYBSUYANX6HZQ62UE2Q, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.647111098 (576h48m27.518194143s) [noise=true , address=145.79.227.12:4767 , piece_id=3LIJHSHFG5R4W6HMB3IUU6PSQK3NQXNFAAIFUIJ5MSAFSWJCU7EQ, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.647442819 (576h48m27.518525865s) [noise=true , address=45.91.116.14:4192 , piece_id=AJ4423NGTBJ27B74XLNBZ5HEXBQ2SGZSLOYZJRD34S6QH3QWZRXA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.647482234 (576h48m27.518565279s) [noise=true , address=145.79.227.42:4121 , piece_id=Q6VZ2FILIGKVM5G3X6A76ZLXQRHGTKW3LCK5AQO5RDZKPP3H7JDA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.647874362 (576h48m27.518957407s) [noise=true , address=77.90.31.17:4773 , piece_id=HJIVFVLP4QMEQTW75OB3ORYSDEB2P3F7SWUY5ULNMKVCVRDQBIZA, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:27.647960545 (576h48m27.519043591s) [noise=true , address=145.79.227.19:4079 , piece_id=WGA23GJV7M2LUN3GZCR6OJ6KKLDE5DORWHUUYDUJYHAFUTOKNJEQ, seg_pos={0 0}, upload=505659]: Uploading piece...
Dec 29 16:17:29.318440773 (576h48m29.189523818s) [upload=505659]: Next returned nil segment
Dec 29 16:17:29.318474677 (576h48m29.189557712s) [upload=505659]: Waiting for error group managing segments...
Dec 29 16:17:29.630061637 (576h48m29.501144682s) [noise=true , address=145.79.227.6:4258 , piece_id=MFITWLD7RZGJATOTKXAY3V3XFXSCCKJPLP3GS3O57GPNF2D6Z3TA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.630698970 (576h48m29.501782005s) [noise=true , address=145.79.227.42:4121 , piece_id=Q6VZ2FILIGKVM5G3X6A76ZLXQRHGTKW3LCK5AQO5RDZKPP3H7JDA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.631699285 (576h48m29.50278232s) [noise=true , address=145.79.227.7:4295 , piece_id=TYS6GDJMD22CNIUGWDWOC25PSXWYYSKXTPFWBTQ6PH2IX6A4SBDA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.632245064 (576h48m29.503328109s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGR3I7DKJWMTF6CTEWAHQA7T2TBGCQZNMPJNKW2UTD5VB6QAEOMA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.632876256 (576h48m29.503959312s) [noise=true , address=145.79.227.48:4303 , piece_id=XUJDEUWVQQUUBZHNH3ZATIV7A5IJEDWEJCCCK7TKGYHPXHTMUJWQ, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.635264914 (576h48m29.506347949s) [noise=true , address=145.79.227.19:4079 , piece_id=WGA23GJV7M2LUN3GZCR6OJ6KKLDE5DORWHUUYDUJYHAFUTOKNJEQ, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.656882338 (576h48m29.527965373s) [noise=true , address=145.79.227.12:4767 , piece_id=3LIJHSHFG5R4W6HMB3IUU6PSQK3NQXNFAAIFUIJ5MSAFSWJCU7EQ, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.822397804 (576h48m29.693480839s) [noise=true , address=77.90.31.21:4877 , piece_id=43UPVJH4H6EXAK2OQZRVXXRPFVYG2QUXBPQLP3KYUMODIP6SG56A, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.939195945 (576h48m29.810278991s) [noise=true , address=77.90.31.14:4661 , piece_id=MUAKYXL5ELANCW3ZJGQP4MRGTKUAD26QVG3ZUZHS7XYCDUBIHIBA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.939698983 (576h48m29.810782018s) [noise=true , address=45.91.116.14:4192 , piece_id=AJ4423NGTBJ27B74XLNBZ5HEXBQ2SGZSLOYZJRD34S6QH3QWZRXA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.940257446 (576h48m29.811340481s) [noise=true , address=77.90.31.19:4841 , piece_id=XZZ4YTHJL24M2PKOOMLX45MMOLBIFH44TRXQT7B4XUL773REC4TQ, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:29.951119279 (576h48m29.822202314s) [noise=true , address=45.91.116.10:4082 , piece_id=UQU2IK2ICBSQ4OVLEK7WNEB3QD55FW7BMG6C2MRPHVVBIFCCZWJQ, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:30.122950663 (576h48m29.994033708s) [noise=true , address=77.90.31.17:4773 , piece_id=HJIVFVLP4QMEQTW75OB3ORYSDEB2P3F7SWUY5ULNMKVCVRDQBIZA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: <nil>
Dec 29 16:17:30.122973455 (576h48m29.9940565s) [seg_pos={0 0}, upload=505659]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:17:30.123217109 (576h48m29.994300144s) [noise=true , address=45.91.116.6:4974 , piece_id=YDACDM2XNQXG2XLQRAG64UTCS5V236PONTFGGWP2ZY6OKFV2P4GA, seg_pos={0 0}, upload=505659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:50490->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:50490->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50490->45.91.116.6:4974: use of closed network connection
Dec 29 16:17:30.123223703 (576h48m29.994306738s) [noise=true , address=45.91.116.6:4974 , piece_id=YDACDM2XNQXG2XLQRAG64UTCS5V236PONTFGGWP2ZY6OKFV2P4GA, seg_pos={0 0}, upload=505659]: Long tail cancellation due to optimal threshold
Dec 29 16:17:30.123306961 (576h48m29.994389996s) [noise=true , address=145.79.227.30:4762 , piece_id=L4U7YR5LLSODCYAAEGO2C6GZV674R4JK6NYBSUYANX6HZQ62UE2Q, seg_pos={0 0}, upload=505659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 29 16:17:30.123322460 (576h48m29.994405495s) [noise=true , address=145.79.227.30:4762 , piece_id=L4U7YR5LLSODCYAAEGO2C6GZV674R4JK6NYBSUYANX6HZQ62UE2Q, seg_pos={0 0}, upload=505659]: Long tail cancellation due to optimal threshold
Dec 29 16:17:30.123561375 (576h48m29.99464441s) [noise=true , address=45.91.116.12:4228 , piece_id=RJW4BARGCY5EF6ASSHP4NGRLVFOBXK4KSG4VB46QOYIFOGNAZA4A, seg_pos={0 0}, upload=505659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:35192->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:35192->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35192->45.91.116.12:4228: use of closed network connection
Dec 29 16:17:30.123570102 (576h48m29.994653137s) [noise=true , address=45.91.116.12:4228 , piece_id=RJW4BARGCY5EF6ASSHP4NGRLVFOBXK4KSG4VB46QOYIFOGNAZA4A, seg_pos={0 0}, upload=505659]: Long tail cancellation due to optimal threshold
Dec 29 16:17:30.123618173 (576h48m29.994701208s) [seg_pos={0 0}, upload=505659]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:17:30.123729605 (576h48m29.99481264s) [upload=505659]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:17:30.123735126 (576h48m29.994818161s) [upload=505659]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:17:30.414929224 (576h48m30.286012259s) [upload=505659]: Done uploading segments...
Dec 29 16:17:30.414948751 (576h48m30.286031786s) [upload=505659]: Upload finished. err: <nil>
Duration: 30.005036988s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
55.737722521s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:17:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 55.737741126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..X] duration:
5.001539328s trace:
https://jaeger.dev.storj.io/trace/53b9d49fea4e448c
FAIL (2025-12-29 16:17:53 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001561561s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX...X.X..XX....XX.......X....X.X......X.....X......X...XX.X] duration:
30.003242294s trace:
https://jaeger.dev.storj.io/trace/2a026f7f3ca97649
FAIL (2025-12-29 16:17:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 4Q3OLLOENKJUVKDQPW37NCUYUPYOAHFKSAWORSI5CZK76EBFFCRQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ILNXV77WJZ527XC76M72ZLC2GBX375WC6C2P4F3Q5IKSRW7GPQ5A) context deadline exceeded
======LOGS======
Dec 29 16:16:48.423894423 (576h47m48.294977468s) [upload=505649]: Starting upload
Dec 29 16:16:48.423989464 (576h47m48.29507255s) [upload=505649]: Done starting upload
Dec 29 16:16:48.424115083 (576h47m48.295198138s) [upload=505649]: Uploading segments...
Dec 29 16:16:48.424271982 (576h47m48.295355017s) [upload=505649]: Got next segment. Inline: false
Dec 29 16:16:48.424341124 (576h47m48.295424179s) [upload=505649]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:16:48.424348478 (576h47m48.295431513s) [upload=505649]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:16:48.673236409 (576h47m48.544319444s) [seg_pos={0 0}, upload=505649]: Begin upload segment...
Dec 29 16:16:48.673281185 (576h47m48.54436422s) [seg_pos={0 0}, upload=505649]: Done begin upload segment.
Dec 29 16:16:48.673303917 (576h47m48.544386952s) [noise=true , address=145.79.227.19:4079 , piece_id=KQXCJ3ZK7WAXUMPSS2DDCNUIA5K5I6MSWJLRZLHANIRVIIIKZKHA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.673344023 (576h47m48.544427068s) [noise=true , address=145.79.227.3:4150 , piece_id=GZ2FV3BTQMN6PZJN2KYC3IUP5BLX44LB2CILG77Y75OE3O7N26BQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.673724518 (576h47m48.544807563s) [noise=true , address=145.79.227.18:4006 , piece_id=DL4FLX7WR7ZH6GPL6SD6RBAWBLQUWX6BR7BW66UCZAOLLUSFH7WA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.674028227 (576h47m48.545111262s) [noise=true , address=145.79.227.17:4971 , piece_id=VIICLSO7FMSOZARL5MQZS54PRV3CXYGTYWB5GRBE56A2AI3HZX7A, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.674292751 (576h47m48.545375786s) [noise=true , address=77.90.31.18:4808 , piece_id=4Q3OLLOENKJUVKDQPW37NCUYUPYOAHFKSAWORSI5CZK76EBFFCRQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.674580057 (576h47m48.545663103s) [noise=true , address=145.79.227.33:4869 , piece_id=UGXO2S32K2V4RTGVR7W3HK4S6B2VLWXRFHOSZD6VAOCVD5ACZ4OA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.674841705 (576h47m48.54592474s) [noise=true , address=145.79.227.37:4566 , piece_id=WEDHIAWI7HMTM5QW33YFBAM7CMAUUWWT4M463TORFKZJFCYUDUJA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.675111519 (576h47m48.546194554s) [noise=true , address=145.79.227.15:4874 , piece_id=RUQTHVWTTC56PEIBA2HP4ZC3KMFYJAHECEN7C3PTTARMF4UUOJ6A, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.675399918 (576h47m48.546482953s) [noise=true , address=45.91.116.7:4012 , piece_id=ILNXV77WJZ527XC76M72ZLC2GBX375WC6C2P4F3Q5IKSRW7GPQ5A, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.676773142 (576h47m48.547856208s) [noise=true , address=145.79.227.47:4266 , piece_id=ABX74EV2ZK4LUDRJ4AMPWY3KRDBFLR63ZDTHIZZI4W267XAV7NBA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.678646881 (576h47m48.549729916s) [noise=true , address=145.79.227.31:4799 , piece_id=NKZXVN3CY2765EF5VUYADB7DATZENXP5UL6WVB2WEEQOR4NNDR3A, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.681186056 (576h47m48.552269091s) [noise=true , address=145.79.227.27:4690 , piece_id=ZYBPO2MNVVWBULO7FHVG2ELR3F5ULG5P5HA3W4CANGX5XJXH4PTQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.683571738 (576h47m48.554654803s) [noise=true , address=145.79.227.13:4802 , piece_id=6JA3IDCPXNYIY425CCAOW5UBLJFFC57YZZTS5CCDC3L25ULE6ACA, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.685582527 (576h47m48.556665592s) [noise=true , address=145.79.227.42:4123 , piece_id=3IY5H6JHM6SYU7O7H4DVMKZAPEKJ4LXEJCCO3E2227HLFYV4M4LQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.686903462 (576h47m48.557986507s) [noise=true , address=145.79.227.24:4582 , piece_id=FTMJVF6FCDB54G3CRS4NMSDV4U6Z366BVJ72MM6NTSCZ2NAN5MUQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:48.688869496 (576h47m48.559952531s) [noise=true , address=145.79.227.52:4446 , piece_id=EW6CZKI5CPQ42BHR6ASGZS7GDDBX6UV52NIWOWFUUKUOCA3LMHSQ, seg_pos={0 0}, upload=505649]: Uploading piece...
Dec 29 16:16:50.373440172 (576h47m50.244523217s) [upload=505649]: Next returned nil segment
Dec 29 16:16:50.373453247 (576h47m50.244536282s) [upload=505649]: Waiting for error group managing segments...
Dec 29 16:16:50.661616012 (576h47m50.532699047s) [noise=true , address=145.79.227.18:4006 , piece_id=DL4FLX7WR7ZH6GPL6SD6RBAWBLQUWX6BR7BW66UCZAOLLUSFH7WA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.668780134 (576h47m50.539863169s) [noise=true , address=145.79.227.52:4446 , piece_id=EW6CZKI5CPQ42BHR6ASGZS7GDDBX6UV52NIWOWFUUKUOCA3LMHSQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.673879174 (576h47m50.544962209s) [noise=true , address=145.79.227.17:4971 , piece_id=VIICLSO7FMSOZARL5MQZS54PRV3CXYGTYWB5GRBE56A2AI3HZX7A, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.674136193 (576h47m50.545219238s) [noise=true , address=145.79.227.15:4874 , piece_id=RUQTHVWTTC56PEIBA2HP4ZC3KMFYJAHECEN7C3PTTARMF4UUOJ6A, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.686038417 (576h47m50.557121452s) [noise=true , address=145.79.227.13:4802 , piece_id=6JA3IDCPXNYIY425CCAOW5UBLJFFC57YZZTS5CCDC3L25ULE6ACA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.687714689 (576h47m50.558797724s) [noise=true , address=145.79.227.37:4566 , piece_id=WEDHIAWI7HMTM5QW33YFBAM7CMAUUWWT4M463TORFKZJFCYUDUJA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.688385636 (576h47m50.559468671s) [noise=true , address=145.79.227.27:4690 , piece_id=ZYBPO2MNVVWBULO7FHVG2ELR3F5ULG5P5HA3W4CANGX5XJXH4PTQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.692936873 (576h47m50.564019918s) [noise=true , address=145.79.227.42:4123 , piece_id=3IY5H6JHM6SYU7O7H4DVMKZAPEKJ4LXEJCCO3E2227HLFYV4M4LQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.695365386 (576h47m50.566448421s) [noise=true , address=145.79.227.47:4266 , piece_id=ABX74EV2ZK4LUDRJ4AMPWY3KRDBFLR63ZDTHIZZI4W267XAV7NBA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.697765486 (576h47m50.568848521s) [noise=true , address=145.79.227.31:4799 , piece_id=NKZXVN3CY2765EF5VUYADB7DATZENXP5UL6WVB2WEEQOR4NNDR3A, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.702231029 (576h47m50.573314064s) [noise=true , address=145.79.227.24:4582 , piece_id=FTMJVF6FCDB54G3CRS4NMSDV4U6Z366BVJ72MM6NTSCZ2NAN5MUQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:50.868310671 (576h47m50.739393706s) [noise=true , address=77.90.31.18:4808 , piece_id=4Q3OLLOENKJUVKDQPW37NCUYUPYOAHFKSAWORSI5CZK76EBFFCRQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:51.958966624 (576h47m51.830049669s) [noise=true , address=45.91.116.7:4012 , piece_id=ILNXV77WJZ527XC76M72ZLC2GBX375WC6C2P4F3Q5IKSRW7GPQ5A, seg_pos={0 0}, upload=505649]: Done uploading piece. err: <nil>
Dec 29 16:16:51.958992001 (576h47m51.830075036s) [seg_pos={0 0}, upload=505649]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:16:51.959256876 (576h47m51.830339911s) [noise=true , address=145.79.227.3:4150 , piece_id=GZ2FV3BTQMN6PZJN2KYC3IUP5BLX44LB2CILG77Y75OE3O7N26BQ, seg_pos={0 0}, upload=505649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF
Dec 29 16:16:51.959266685 (576h47m51.83034972s) [noise=true , address=145.79.227.3:4150 , piece_id=GZ2FV3BTQMN6PZJN2KYC3IUP5BLX44LB2CILG77Y75OE3O7N26BQ, seg_pos={0 0}, upload=505649]: Long tail cancellation due to optimal threshold
Dec 29 16:16:51.959535276 (576h47m51.830618311s) [noise=true , address=145.79.227.33:4869 , piece_id=UGXO2S32K2V4RTGVR7W3HK4S6B2VLWXRFHOSZD6VAOCVD5ACZ4OA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:16:51.959547068 (576h47m51.830630114s) [noise=true , address=145.79.227.33:4869 , piece_id=UGXO2S32K2V4RTGVR7W3HK4S6B2VLWXRFHOSZD6VAOCVD5ACZ4OA, seg_pos={0 0}, upload=505649]: Long tail cancellation due to optimal threshold
Dec 29 16:16:51.959666977 (576h47m51.830750012s) [noise=true , address=145.79.227.19:4079 , piece_id=KQXCJ3ZK7WAXUMPSS2DDCNUIA5K5I6MSWJLRZLHANIRVIIIKZKHA, seg_pos={0 0}, upload=505649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF
Dec 29 16:16:51.959676505 (576h47m51.83075955s) [noise=true , address=145.79.227.19:4079 , piece_id=KQXCJ3ZK7WAXUMPSS2DDCNUIA5K5I6MSWJLRZLHANIRVIIIKZKHA, seg_pos={0 0}, upload=505649]: Long tail cancellation due to optimal threshold
Dec 29 16:16:51.959695883 (576h47m51.830778928s) [seg_pos={0 0}, upload=505649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:16:51.959787005 (576h47m51.83087005s) [upload=505649]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:16:51.959794720 (576h47m51.830877765s) [upload=505649]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:16:52.257239420 (576h47m52.128322455s) [upload=505649]: Done uploading segments...
Dec 29 16:16:52.257263426 (576h47m52.128346471s) [upload=505649]: Upload finished. err: <nil>
Duration: 30.003263844s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
15.605831251s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:17:15 UTC)
orbiter warning: context deadline exceeded
Duration: 15.605869645s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
40.276923659s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:17:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 40.276949026s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..] duration:
5.001268031s trace:
https://jaeger.dev.storj.io/trace/42f366d0fe41757
FAIL (2025-12-29 16:16:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001292417s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX...X.X..XX....XX.......X....X.X......X.....X......X...XX.] duration:
6.98232204s trace:
https://jaeger.dev.storj.io/trace/6025b27f17334397
FAIL (2025-12-29 16:16:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:15:45.264551509 (576h46m45.135634544s) [upload=505632]: Starting upload
Dec 29 16:15:45.264635839 (576h46m45.135718894s) [upload=505632]: Done starting upload
Dec 29 16:15:45.264801104 (576h46m45.135884139s) [upload=505632]: Uploading segments...
Dec 29 16:15:45.265077470 (576h46m45.136160515s) [upload=505632]: Got next segment. Inline: false
Dec 29 16:15:45.265132435 (576h46m45.13621547s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:15:45.265153886 (576h46m45.136236941s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:15:45.511233148 (576h46m45.382316193s) [seg_pos={0 0}, upload=505632]: Begin upload segment...
Dec 29 16:15:45.511280228 (576h46m45.382363263s) [seg_pos={0 0}, upload=505632]: Done begin upload segment.
Dec 29 16:15:45.511308672 (576h46m45.382391717s) [noise=true , address=145.79.227.15:4876 , piece_id=S64GGAW5GVD54YYCFOQ5MIIQQA7TELDZE4TSUDQVXFMZVL3OIHTQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.511321777 (576h46m45.382404812s) [noise=true , address=145.79.227.36:4518 , piece_id=ZOEQ27OM7TJGJIQY6D2GJSOCNGNLVKLOHDFR2TA4NB2SS5HOBZJA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.511797312 (576h46m45.382880357s) [noise=true , address=145.79.227.19:4077 , piece_id=KE44TBPWMD2LDX3V33ZC642YWY22YCUQ2HK45CM2BGAUT2N7MJNA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512181754 (576h46m45.383264799s) [noise=true , address=45.91.116.6:4973 , piece_id=C6UNMC5WEWMHR74QN6OTQEODJWEPQFKE4T5IZZ7P3MZ2ULVJJLTQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512576677 (576h46m45.383659722s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512977850 (576h46m45.384060885s) [noise=true , address=145.79.227.48:4302 , piece_id=QRANMPVULHWOL6JUXVOFYBNIOAIH5HIGJS3NB7DMZKSSD66PBPLQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.513456752 (576h46m45.384539787s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.513875279 (576h46m45.384958324s) [noise=true , address=145.79.227.47:4267 , piece_id=7UCU2AUL6V7MWTWEVLVLLMJSX7QGVFEVC3LR7HTQCSJUB3ZRHMRQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.514295980 (576h46m45.385379015s) [noise=true , address=45.91.116.8:4045 , piece_id=GUHH3GRUWHCDBKREGGCPVRYNYRM3UHCJN7ML2VCXVOF7MITDOELQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.514667548 (576h46m45.385750583s) [noise=true , address=145.79.227.28:4726 , piece_id=JG2FPGHMFDDSXY2OQ5UH6D3ZMANXWAKBNFYWXMXSAYFJHTPU5FNA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.515914101 (576h46m45.386997146s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.517433825 (576h46m45.38851687s) [noise=true , address=45.91.116.5:4300 , piece_id=7NBG4UOIFVFX7RITPPU3P2AOXFVQSQRWD57KVYFVIXJH4U6RU3MA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.518647787 (576h46m45.389730832s) [noise=true , address=145.79.227.11:4732 , piece_id=KLMEPTWPOGWF22B4TQ2ADAP7XQPE6BRC2BFWIKOIOJCBG6MCZQGQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.520060607 (576h46m45.391143653s) [noise=true , address=145.79.227.37:4566 , piece_id=JYOTC4TEMYE5S7IAOK4Q7NOT4B4FIU76ZMNPNCDEN2PF6CRAD36Q, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.521369559 (576h46m45.392452604s) [noise=true , address=145.79.227.16:4909 , piece_id=FDTYQQ5FC3MAHHY6B4Q7MNRSJQXL3UAT3H7PLNZQFBBIVMGXI5KA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.522638486 (576h46m45.393721531s) [noise=true , address=145.79.227.12:4765 , piece_id=2SL2Y5LGDWB7XNRCR7L2LGSY4CG5KOVDE6FFFKQRCBZ47CXEWT5Q, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:47.103432187 (576h46m46.974515222s) [upload=505632]: Next returned nil segment
Dec 29 16:15:47.103443959 (576h46m46.974526994s) [upload=505632]: Waiting for error group managing segments...
Dec 29 16:15:47.482115467 (576h46m47.353198512s) [noise=true , address=145.79.227.19:4077 , piece_id=KE44TBPWMD2LDX3V33ZC642YWY22YCUQ2HK45CM2BGAUT2N7MJNA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.486419914 (576h46m47.357502949s) [noise=true , address=145.79.227.48:4302 , piece_id=QRANMPVULHWOL6JUXVOFYBNIOAIH5HIGJS3NB7DMZKSSD66PBPLQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.493727598 (576h46m47.364810643s) [noise=true , address=145.79.227.15:4876 , piece_id=S64GGAW5GVD54YYCFOQ5MIIQQA7TELDZE4TSUDQVXFMZVL3OIHTQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.493998144 (576h46m47.365081179s) [noise=true , address=145.79.227.37:4566 , piece_id=JYOTC4TEMYE5S7IAOK4Q7NOT4B4FIU76ZMNPNCDEN2PF6CRAD36Q, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.500132314 (576h46m47.371215359s) [noise=true , address=145.79.227.28:4726 , piece_id=JG2FPGHMFDDSXY2OQ5UH6D3ZMANXWAKBNFYWXMXSAYFJHTPU5FNA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.510735103 (576h46m47.381818138s) [noise=true , address=145.79.227.47:4267 , piece_id=7UCU2AUL6V7MWTWEVLVLLMJSX7QGVFEVC3LR7HTQCSJUB3ZRHMRQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.521157910 (576h46m47.392240945s) [noise=true , address=145.79.227.11:4732 , piece_id=KLMEPTWPOGWF22B4TQ2ADAP7XQPE6BRC2BFWIKOIOJCBG6MCZQGQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.532503664 (576h46m47.403586709s) [noise=true , address=145.79.227.12:4765 , piece_id=2SL2Y5LGDWB7XNRCR7L2LGSY4CG5KOVDE6FFFKQRCBZ47CXEWT5Q, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.539739101 (576h46m47.410822146s) [noise=true , address=145.79.227.16:4909 , piece_id=FDTYQQ5FC3MAHHY6B4Q7MNRSJQXL3UAT3H7PLNZQFBBIVMGXI5KA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.540067476 (576h46m47.411150521s) [noise=true , address=145.79.227.36:4518 , piece_id=ZOEQ27OM7TJGJIQY6D2GJSOCNGNLVKLOHDFR2TA4NB2SS5HOBZJA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.770211095 (576h46m47.64129414s) [noise=true , address=45.91.116.6:4973 , piece_id=C6UNMC5WEWMHR74QN6OTQEODJWEPQFKE4T5IZZ7P3MZ2ULVJJLTQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.891203573 (576h46m47.762286618s) [noise=true , address=45.91.116.8:4045 , piece_id=GUHH3GRUWHCDBKREGGCPVRYNYRM3UHCJN7ML2VCXVOF7MITDOELQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:48.234359344 (576h46m48.105442389s) [noise=true , address=45.91.116.5:4300 , piece_id=7NBG4UOIFVFX7RITPPU3P2AOXFVQSQRWD57KVYFVIXJH4U6RU3MA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:48.234379643 (576h46m48.105462678s) [seg_pos={0 0}, upload=505632]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:15:48.234571618 (576h46m48.105654653s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234581637 (576h46m48.105664682s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234806085 (576h46m48.10588912s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234811705 (576h46m48.10589475s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234906095 (576h46m48.10598913s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234911035 (576h46m48.10599407s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234924951 (576h46m48.106007986s) [seg_pos={0 0}, upload=505632]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:15:48.234989824 (576h46m48.106072859s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:15:48.234994703 (576h46m48.106077739s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:15:48.535486706 (576h46m48.406569751s) [upload=505632]: Done uploading segments...
Dec 29 16:15:48.535508478 (576h46m48.406591523s) [upload=505632]: Upload finished. err: <nil>
Duration: 6.982355513s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
12.156007791s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:16:26 UTC)
orbiter warning: context deadline exceeded
Duration: 12.156035585s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.XX...X.X..XX....XX.......X....X.X......X.....X......X...XX] duration:
30.003266638s trace:
https://jaeger.dev.storj.io/trace/6025b27f17334397
FAIL (2025-12-29 16:16:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:15:45.264551509 (576h46m45.135634544s) [upload=505632]: Starting upload
Dec 29 16:15:45.264635839 (576h46m45.135718894s) [upload=505632]: Done starting upload
Dec 29 16:15:45.264801104 (576h46m45.135884139s) [upload=505632]: Uploading segments...
Dec 29 16:15:45.265077470 (576h46m45.136160515s) [upload=505632]: Got next segment. Inline: false
Dec 29 16:15:45.265132435 (576h46m45.13621547s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:15:45.265153886 (576h46m45.136236941s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:15:45.511233148 (576h46m45.382316193s) [seg_pos={0 0}, upload=505632]: Begin upload segment...
Dec 29 16:15:45.511280228 (576h46m45.382363263s) [seg_pos={0 0}, upload=505632]: Done begin upload segment.
Dec 29 16:15:45.511308672 (576h46m45.382391717s) [noise=true , address=145.79.227.15:4876 , piece_id=S64GGAW5GVD54YYCFOQ5MIIQQA7TELDZE4TSUDQVXFMZVL3OIHTQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.511321777 (576h46m45.382404812s) [noise=true , address=145.79.227.36:4518 , piece_id=ZOEQ27OM7TJGJIQY6D2GJSOCNGNLVKLOHDFR2TA4NB2SS5HOBZJA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.511797312 (576h46m45.382880357s) [noise=true , address=145.79.227.19:4077 , piece_id=KE44TBPWMD2LDX3V33ZC642YWY22YCUQ2HK45CM2BGAUT2N7MJNA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512181754 (576h46m45.383264799s) [noise=true , address=45.91.116.6:4973 , piece_id=C6UNMC5WEWMHR74QN6OTQEODJWEPQFKE4T5IZZ7P3MZ2ULVJJLTQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512576677 (576h46m45.383659722s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.512977850 (576h46m45.384060885s) [noise=true , address=145.79.227.48:4302 , piece_id=QRANMPVULHWOL6JUXVOFYBNIOAIH5HIGJS3NB7DMZKSSD66PBPLQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.513456752 (576h46m45.384539787s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.513875279 (576h46m45.384958324s) [noise=true , address=145.79.227.47:4267 , piece_id=7UCU2AUL6V7MWTWEVLVLLMJSX7QGVFEVC3LR7HTQCSJUB3ZRHMRQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.514295980 (576h46m45.385379015s) [noise=true , address=45.91.116.8:4045 , piece_id=GUHH3GRUWHCDBKREGGCPVRYNYRM3UHCJN7ML2VCXVOF7MITDOELQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.514667548 (576h46m45.385750583s) [noise=true , address=145.79.227.28:4726 , piece_id=JG2FPGHMFDDSXY2OQ5UH6D3ZMANXWAKBNFYWXMXSAYFJHTPU5FNA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.515914101 (576h46m45.386997146s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.517433825 (576h46m45.38851687s) [noise=true , address=45.91.116.5:4300 , piece_id=7NBG4UOIFVFX7RITPPU3P2AOXFVQSQRWD57KVYFVIXJH4U6RU3MA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.518647787 (576h46m45.389730832s) [noise=true , address=145.79.227.11:4732 , piece_id=KLMEPTWPOGWF22B4TQ2ADAP7XQPE6BRC2BFWIKOIOJCBG6MCZQGQ, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.520060607 (576h46m45.391143653s) [noise=true , address=145.79.227.37:4566 , piece_id=JYOTC4TEMYE5S7IAOK4Q7NOT4B4FIU76ZMNPNCDEN2PF6CRAD36Q, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.521369559 (576h46m45.392452604s) [noise=true , address=145.79.227.16:4909 , piece_id=FDTYQQ5FC3MAHHY6B4Q7MNRSJQXL3UAT3H7PLNZQFBBIVMGXI5KA, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:45.522638486 (576h46m45.393721531s) [noise=true , address=145.79.227.12:4765 , piece_id=2SL2Y5LGDWB7XNRCR7L2LGSY4CG5KOVDE6FFFKQRCBZ47CXEWT5Q, seg_pos={0 0}, upload=505632]: Uploading piece...
Dec 29 16:15:47.103432187 (576h46m46.974515222s) [upload=505632]: Next returned nil segment
Dec 29 16:15:47.103443959 (576h46m46.974526994s) [upload=505632]: Waiting for error group managing segments...
Dec 29 16:15:47.482115467 (576h46m47.353198512s) [noise=true , address=145.79.227.19:4077 , piece_id=KE44TBPWMD2LDX3V33ZC642YWY22YCUQ2HK45CM2BGAUT2N7MJNA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.486419914 (576h46m47.357502949s) [noise=true , address=145.79.227.48:4302 , piece_id=QRANMPVULHWOL6JUXVOFYBNIOAIH5HIGJS3NB7DMZKSSD66PBPLQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.493727598 (576h46m47.364810643s) [noise=true , address=145.79.227.15:4876 , piece_id=S64GGAW5GVD54YYCFOQ5MIIQQA7TELDZE4TSUDQVXFMZVL3OIHTQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.493998144 (576h46m47.365081179s) [noise=true , address=145.79.227.37:4566 , piece_id=JYOTC4TEMYE5S7IAOK4Q7NOT4B4FIU76ZMNPNCDEN2PF6CRAD36Q, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.500132314 (576h46m47.371215359s) [noise=true , address=145.79.227.28:4726 , piece_id=JG2FPGHMFDDSXY2OQ5UH6D3ZMANXWAKBNFYWXMXSAYFJHTPU5FNA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.510735103 (576h46m47.381818138s) [noise=true , address=145.79.227.47:4267 , piece_id=7UCU2AUL6V7MWTWEVLVLLMJSX7QGVFEVC3LR7HTQCSJUB3ZRHMRQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.521157910 (576h46m47.392240945s) [noise=true , address=145.79.227.11:4732 , piece_id=KLMEPTWPOGWF22B4TQ2ADAP7XQPE6BRC2BFWIKOIOJCBG6MCZQGQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.532503664 (576h46m47.403586709s) [noise=true , address=145.79.227.12:4765 , piece_id=2SL2Y5LGDWB7XNRCR7L2LGSY4CG5KOVDE6FFFKQRCBZ47CXEWT5Q, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.539739101 (576h46m47.410822146s) [noise=true , address=145.79.227.16:4909 , piece_id=FDTYQQ5FC3MAHHY6B4Q7MNRSJQXL3UAT3H7PLNZQFBBIVMGXI5KA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.540067476 (576h46m47.411150521s) [noise=true , address=145.79.227.36:4518 , piece_id=ZOEQ27OM7TJGJIQY6D2GJSOCNGNLVKLOHDFR2TA4NB2SS5HOBZJA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.770211095 (576h46m47.64129414s) [noise=true , address=45.91.116.6:4973 , piece_id=C6UNMC5WEWMHR74QN6OTQEODJWEPQFKE4T5IZZ7P3MZ2ULVJJLTQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:47.891203573 (576h46m47.762286618s) [noise=true , address=45.91.116.8:4045 , piece_id=GUHH3GRUWHCDBKREGGCPVRYNYRM3UHCJN7ML2VCXVOF7MITDOELQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:48.234359344 (576h46m48.105442389s) [noise=true , address=45.91.116.5:4300 , piece_id=7NBG4UOIFVFX7RITPPU3P2AOXFVQSQRWD57KVYFVIXJH4U6RU3MA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: <nil>
Dec 29 16:15:48.234379643 (576h46m48.105462678s) [seg_pos={0 0}, upload=505632]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:15:48.234571618 (576h46m48.105654653s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234581637 (576h46m48.105664682s) [noise=true , address=45.91.116.9:4264 , piece_id=EKWOKTL5TCGFMSKP2KXXCR6KOK7APR5GXMEX5GKRTZ5ZFGOH2DYQ, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234806085 (576h46m48.10588912s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234811705 (576h46m48.10589475s) [noise=true , address=45.91.116.14:4191 , piece_id=X6VYXS32N7CWYOU5JXUCF7LKY3437WZGBPMHRGZILD7PBRLN6YPA, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234906095 (576h46m48.10598913s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:48.234911035 (576h46m48.10599407s) [noise=true , address=77.90.31.19:4841 , piece_id=AD3YNEADNO2UWVLYY4RWXFTCAW4CPR2QW4XBST5GR4ZWZFAUGJYA, seg_pos={0 0}, upload=505632]: Long tail cancellation due to optimal threshold
Dec 29 16:15:48.234924951 (576h46m48.106007986s) [seg_pos={0 0}, upload=505632]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:15:48.234989824 (576h46m48.106072859s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:15:48.234994703 (576h46m48.106077739s) [upload=505632]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:15:48.535486706 (576h46m48.406569751s) [upload=505632]: Done uploading segments...
Dec 29 16:15:48.535508478 (576h46m48.406591523s) [upload=505632]: Upload finished. err: <nil>
Duration: 30.003282918s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
44.690961934s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:16:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 44.690992293s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX.] duration:
5.00105228s trace:
https://jaeger.dev.storj.io/trace/42f366d0fe41757
FAIL (2025-12-29 16:16:00 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001081295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX.XX...X.X..XX....XX.......X....X.X......X.....X......X...X] duration:
30.002676518s trace:
https://jaeger.dev.storj.io/trace/1538a2ef439d2851
FAIL (2025-12-29 16:15:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AEQCHYJIAFNMUYYHI3RMBWHO44U46D5VR6X5ZQHBWNYWO3V6CCOQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YIU7JQH36DUH6VQXJ72B6GDTUWGSSTFVWKAEIXALSDAXZSFCXSIQ) context deadline exceeded
======LOGS======
Dec 29 16:15:03.786079614 (576h46m3.657162659s) [upload=505622]: Starting upload
Dec 29 16:15:03.786175295 (576h46m3.65725834s) [upload=505622]: Done starting upload
Dec 29 16:15:03.786398140 (576h46m3.657481195s) [upload=505622]: Uploading segments...
Dec 29 16:15:03.786526183 (576h46m3.657609228s) [upload=505622]: Got next segment. Inline: false
Dec 29 16:15:03.786595976 (576h46m3.657679021s) [upload=505622]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:15:03.786603080 (576h46m3.657686125s) [upload=505622]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:15:04.036024246 (576h46m3.907107301s) [seg_pos={0 0}, upload=505622]: Begin upload segment...
Dec 29 16:15:04.036091324 (576h46m3.907174379s) [seg_pos={0 0}, upload=505622]: Done begin upload segment.
Dec 29 16:15:04.036160065 (576h46m3.90724311s) [noise=true , address=145.79.227.38:4977 , piece_id=M3VF6EWCBE2TRLZZRO2A7BZDTKY6URGAAB3XACRV76XVRGB7EY7Q, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.036204910 (576h46m3.907287995s) [noise=true , address=145.79.227.9:4365 , piece_id=I4TKZEUL3HOI5VIMNNH3HSUAOSJWCHWBJKPGMSC5NMPXKDG52XEA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.037083012 (576h46m3.908166057s) [noise=true , address=145.79.227.4:4185 , piece_id=UCABKGIQGNOS3HBZCZNWRBSRJPQ2UINUBHTIX53BY3ALEXKTAHNQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.038001111 (576h46m3.909084156s) [noise=true , address=77.90.31.15:4700 , piece_id=YIU7JQH36DUH6VQXJ72B6GDTUWGSSTFVWKAEIXALSDAXZSFCXSIQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.038753533 (576h46m3.909836578s) [noise=true , address=45.91.116.9:4262 , piece_id=AEQCHYJIAFNMUYYHI3RMBWHO44U46D5VR6X5ZQHBWNYWO3V6CCOQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.038850878 (576h46m3.909933923s) [noise=true , address=145.79.227.34:4906 , piece_id=IJ6LR7FWR7TUYAZZDMMYNJ5NKXBBGVDVBXZUOBFZUEEWZO2CAWCQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.040036326 (576h46m3.911119371s) [noise=true , address=77.90.31.13:4629 , piece_id=KFB2ZFMW7SEDOJOBUKG27Q6EAEPX37G54XKL3YU5UM2NN7IVBAGQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.040121758 (576h46m3.911204803s) [noise=true , address=145.79.227.52:4445 , piece_id=7NZVATSEOFG554JW5MGCA5IQU676545CNBEE67DB3RQS2XG6LDEA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.041489313 (576h46m3.912572358s) [noise=true , address=145.79.227.42:4122 , piece_id=CHSZAPKIH7RRHJMXHC6FUNZKIPK5NXRQBGWMCKYZH7SI2B6LSNMA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.042134370 (576h46m3.913217415s) [noise=true , address=45.91.116.8:4046 , piece_id=QETNWGCS24M54RCNNQICYJ3MWXH327AVYSDS2LDYUR74CLJTVDMA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.042219283 (576h46m3.913302328s) [noise=true , address=145.79.227.39:4014 , piece_id=B6ZDJW3Z6HD44ISFBMQZG5K5CG6AU6YH36VDMPINHK2L2OU5UKCQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.042971835 (576h46m3.91405488s) [noise=true , address=145.79.227.11:4730 , piece_id=BSXRTP7UVVKPM5BWIPRDNYGTT2THE2HVLCI36BKJCKSLJSIX72EQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.043520811 (576h46m3.914603856s) [noise=true , address=77.90.31.12:4589 , piece_id=ILCV4OQI7NAQ3U32VR7XBUCEL7MRO7CX2MUQSH2APHECJWPSTYWA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.043985024 (576h46m3.915068069s) [noise=true , address=145.79.227.19:4078 , piece_id=L2DCB6OLL2X5HWSOIAU6RYSAM5YZOOWBLIAL66GPSK4EPGPLV7HA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.046372120 (576h46m3.917455165s) [noise=true , address=45.91.116.13:4154 , piece_id=LIH2Y6BVNJ47IOWDYDWFPGV2YW5FUZ2T5RIUD5VRCVQZI4QRAKSQ, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:04.048217193 (576h46m3.919300238s) [noise=true , address=45.91.116.10:4082 , piece_id=DBZQOQIMKRWAGNXNML2GNWXKJ667KSJIYMETNURKT6EXLGAFIJSA, seg_pos={0 0}, upload=505622]: Uploading piece...
Dec 29 16:15:05.747378363 (576h46m5.618461398s) [upload=505622]: Next returned nil segment
Dec 29 16:15:05.747393502 (576h46m5.618476557s) [upload=505622]: Waiting for error group managing segments...
Dec 29 16:15:06.032240955 (576h46m5.903324s) [noise=true , address=145.79.227.9:4365 , piece_id=I4TKZEUL3HOI5VIMNNH3HSUAOSJWCHWBJKPGMSC5NMPXKDG52XEA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.039205667 (576h46m5.910288732s) [noise=true , address=145.79.227.52:4445 , piece_id=7NZVATSEOFG554JW5MGCA5IQU676545CNBEE67DB3RQS2XG6LDEA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.050457802 (576h46m5.921540847s) [noise=true , address=145.79.227.42:4122 , piece_id=CHSZAPKIH7RRHJMXHC6FUNZKIPK5NXRQBGWMCKYZH7SI2B6LSNMA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.052235006 (576h46m5.923318041s) [noise=true , address=145.79.227.4:4185 , piece_id=UCABKGIQGNOS3HBZCZNWRBSRJPQ2UINUBHTIX53BY3ALEXKTAHNQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.056841648 (576h46m5.927924693s) [noise=true , address=145.79.227.11:4730 , piece_id=BSXRTP7UVVKPM5BWIPRDNYGTT2THE2HVLCI36BKJCKSLJSIX72EQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.058542928 (576h46m5.929625973s) [noise=true , address=145.79.227.38:4977 , piece_id=M3VF6EWCBE2TRLZZRO2A7BZDTKY6URGAAB3XACRV76XVRGB7EY7Q, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.059591654 (576h46m5.930674699s) [noise=true , address=145.79.227.39:4014 , piece_id=B6ZDJW3Z6HD44ISFBMQZG5K5CG6AU6YH36VDMPINHK2L2OU5UKCQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.074300151 (576h46m5.945383227s) [noise=true , address=145.79.227.19:4078 , piece_id=L2DCB6OLL2X5HWSOIAU6RYSAM5YZOOWBLIAL66GPSK4EPGPLV7HA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.235869875 (576h46m6.10695291s) [noise=true , address=77.90.31.13:4629 , piece_id=KFB2ZFMW7SEDOJOBUKG27Q6EAEPX37G54XKL3YU5UM2NN7IVBAGQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.345596661 (576h46m6.216679706s) [noise=true , address=45.91.116.10:4082 , piece_id=DBZQOQIMKRWAGNXNML2GNWXKJ667KSJIYMETNURKT6EXLGAFIJSA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.355382014 (576h46m6.226465049s) [noise=true , address=77.90.31.12:4589 , piece_id=ILCV4OQI7NAQ3U32VR7XBUCEL7MRO7CX2MUQSH2APHECJWPSTYWA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:06.484683573 (576h46m6.355766618s) [noise=true , address=45.91.116.9:4262 , piece_id=AEQCHYJIAFNMUYYHI3RMBWHO44U46D5VR6X5ZQHBWNYWO3V6CCOQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:07.078268430 (576h46m6.949351475s) [noise=true , address=77.90.31.15:4700 , piece_id=YIU7JQH36DUH6VQXJ72B6GDTUWGSSTFVWKAEIXALSDAXZSFCXSIQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: <nil>
Dec 29 16:15:07.078290351 (576h46m6.949373386s) [seg_pos={0 0}, upload=505622]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:15:07.078512745 (576h46m6.94959578s) [noise=true , address=45.91.116.13:4154 , piece_id=LIH2Y6BVNJ47IOWDYDWFPGV2YW5FUZ2T5RIUD5VRCVQZI4QRAKSQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:07.078519728 (576h46m6.949602773s) [noise=true , address=45.91.116.13:4154 , piece_id=LIH2Y6BVNJ47IOWDYDWFPGV2YW5FUZ2T5RIUD5VRCVQZI4QRAKSQ, seg_pos={0 0}, upload=505622]: Long tail cancellation due to optimal threshold
Dec 29 16:15:07.078777809 (576h46m6.949860854s) [noise=true , address=145.79.227.34:4906 , piece_id=IJ6LR7FWR7TUYAZZDMMYNJ5NKXBBGVDVBXZUOBFZUEEWZO2CAWCQ, seg_pos={0 0}, upload=505622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:07.078785354 (576h46m6.949868389s) [noise=true , address=145.79.227.34:4906 , piece_id=IJ6LR7FWR7TUYAZZDMMYNJ5NKXBBGVDVBXZUOBFZUEEWZO2CAWCQ, seg_pos={0 0}, upload=505622]: Long tail cancellation due to optimal threshold
Dec 29 16:15:07.078800312 (576h46m6.949883357s) [noise=true , address=45.91.116.8:4046 , piece_id=QETNWGCS24M54RCNNQICYJ3MWXH327AVYSDS2LDYUR74CLJTVDMA, seg_pos={0 0}, upload=505622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 29 16:15:07.078819589 (576h46m6.949902624s) [noise=true , address=45.91.116.8:4046 , piece_id=QETNWGCS24M54RCNNQICYJ3MWXH327AVYSDS2LDYUR74CLJTVDMA, seg_pos={0 0}, upload=505622]: Long tail cancellation due to optimal threshold
Dec 29 16:15:07.078845788 (576h46m6.949928833s) [seg_pos={0 0}, upload=505622]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:15:07.078950548 (576h46m6.950033593s) [upload=505622]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:15:07.078956891 (576h46m6.950039936s) [upload=505622]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:15:07.390482693 (576h46m7.261565728s) [upload=505622]: Done uploading segments...
Dec 29 16:15:07.390501348 (576h46m7.261584383s) [upload=505622]: Upload finished. err: <nil>
Duration: 30.002705014s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
19.606404393s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:15:27 UTC)
orbiter warning: context deadline exceeded
Duration: 19.606435031s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX] duration:
5.001291878s trace:
https://jaeger.dev.storj.io/trace/42f366d0fe41757
FAIL (2025-12-29 16:15:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001316544s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
44.658617152s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:15:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 44.658640546s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.X] duration:
5.000921811s trace:
https://jaeger.dev.storj.io/trace/5bb5bd9d4d521881
FAIL (2025-12-29 16:15:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001155324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................XX.........................................] duration:
52.668547022s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:14:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 52.668572863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
7.886626212s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:14:25 UTC)
orbiter warning: context deadline exceeded
Duration: 7.886660547s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.] duration:
5.001629606s trace:
https://jaeger.dev.storj.io/trace/79c2f6262ab8dfd2
FAIL (2025-12-29 16:14:23 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00171031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX.XX...X.X..XX....XX.......X....X.X......X.....X......X...] duration:
6.030106674s trace:
https://jaeger.dev.storj.io/trace/35f0948d120e1a64
FAIL (2025-12-29 16:14:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:11:01.540027575 (576h42m1.41111063s) [upload=505569]: Starting upload
Dec 29 16:11:01.540125631 (576h42m1.411208677s) [upload=505569]: Done starting upload
Dec 29 16:11:01.540172460 (576h42m1.411255515s) [upload=505569]: Uploading segments...
Dec 29 16:11:01.540402268 (576h42m1.411485313s) [upload=505569]: Got next segment. Inline: false
Dec 29 16:11:01.540453786 (576h42m1.411536831s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:11:01.540459968 (576h42m1.411543013s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:11:01.796687083 (576h42m1.667770138s) [seg_pos={0 0}, upload=505569]: Begin upload segment...
Dec 29 16:11:01.796746415 (576h42m1.66782946s) [seg_pos={0 0}, upload=505569]: Done begin upload segment.
Dec 29 16:11:01.796782695 (576h42m1.66786574s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797236830 (576h42m1.668319885s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797759034 (576h42m1.668842079s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797992658 (576h42m1.669075743s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798437265 (576h42m1.66952031s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798865170 (576h42m1.669948215s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799287043 (576h42m1.670370098s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799724426 (576h42m1.670807471s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800158061 (576h42m1.671241106s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800580416 (576h42m1.671663461s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801347767 (576h42m1.672430813s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801878428 (576h42m1.672961473s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802324087 (576h42m1.673407142s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802762020 (576h42m1.673845065s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803196748 (576h42m1.674279793s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803644912 (576h42m1.674727957s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:03.406181393 (576h42m3.277264448s) [upload=505569]: Next returned nil segment
Dec 29 16:11:03.406199919 (576h42m3.277282964s) [upload=505569]: Waiting for error group managing segments...
Dec 29 16:11:03.790763095 (576h42m3.66184614s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.790929141 (576h42m3.662012186s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.796025666 (576h42m3.667108711s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.799086945 (576h42m3.67016999s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.802427456 (576h42m3.673510501s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.803905590 (576h42m3.674988635s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.805330424 (576h42m3.676413469s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.812366111 (576h42m3.683449156s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.813050002 (576h42m3.684133047s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.065913082 (576h42m3.936996127s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.179797156 (576h42m4.050880221s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.231768326 (576h42m4.102851371s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456490982 (576h42m4.327574027s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456512182 (576h42m4.327595227s) [seg_pos={0 0}, upload=505569]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:11:04.456753241 (576h42m4.327836276s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.456760115 (576h42m4.32784316s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457005642 (576h42m4.328088687s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457012916 (576h42m4.328095961s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457082919 (576h42m4.328165964s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457120811 (576h42m4.328203846s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457152792 (576h42m4.328235837s) [seg_pos={0 0}, upload=505569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:11:04.457255488 (576h42m4.328338533s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:11:04.457262581 (576h42m4.328345616s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:11:04.748221661 (576h42m4.619304706s) [upload=505569]: Done uploading segments...
Dec 29 16:11:04.748245086 (576h42m4.619328131s) [upload=505569]: Upload finished. err: <nil>
Duration: 6.030137483s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X] duration:
5.001017176s trace:
https://jaeger.dev.storj.io/trace/79c2f6262ab8dfd2
FAIL (2025-12-29 16:14:04 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001040961s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................XX........................................] duration:
45.411643875s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:13:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 45.411673802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
7.309029991s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:13:34 UTC)
orbiter warning: context deadline exceeded
Duration: 7.30906075s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX.XX...X.X..XX....XX.......X....X.X......X.....X......X..] duration:
6.043312925s trace:
https://jaeger.dev.storj.io/trace/35f0948d120e1a64
FAIL (2025-12-29 16:13:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:11:01.540027575 (576h42m1.41111063s) [upload=505569]: Starting upload
Dec 29 16:11:01.540125631 (576h42m1.411208677s) [upload=505569]: Done starting upload
Dec 29 16:11:01.540172460 (576h42m1.411255515s) [upload=505569]: Uploading segments...
Dec 29 16:11:01.540402268 (576h42m1.411485313s) [upload=505569]: Got next segment. Inline: false
Dec 29 16:11:01.540453786 (576h42m1.411536831s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:11:01.540459968 (576h42m1.411543013s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:11:01.796687083 (576h42m1.667770138s) [seg_pos={0 0}, upload=505569]: Begin upload segment...
Dec 29 16:11:01.796746415 (576h42m1.66782946s) [seg_pos={0 0}, upload=505569]: Done begin upload segment.
Dec 29 16:11:01.796782695 (576h42m1.66786574s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797236830 (576h42m1.668319885s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797759034 (576h42m1.668842079s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797992658 (576h42m1.669075743s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798437265 (576h42m1.66952031s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798865170 (576h42m1.669948215s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799287043 (576h42m1.670370098s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799724426 (576h42m1.670807471s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800158061 (576h42m1.671241106s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800580416 (576h42m1.671663461s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801347767 (576h42m1.672430813s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801878428 (576h42m1.672961473s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802324087 (576h42m1.673407142s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802762020 (576h42m1.673845065s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803196748 (576h42m1.674279793s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803644912 (576h42m1.674727957s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:03.406181393 (576h42m3.277264448s) [upload=505569]: Next returned nil segment
Dec 29 16:11:03.406199919 (576h42m3.277282964s) [upload=505569]: Waiting for error group managing segments...
Dec 29 16:11:03.790763095 (576h42m3.66184614s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.790929141 (576h42m3.662012186s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.796025666 (576h42m3.667108711s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.799086945 (576h42m3.67016999s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.802427456 (576h42m3.673510501s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.803905590 (576h42m3.674988635s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.805330424 (576h42m3.676413469s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.812366111 (576h42m3.683449156s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.813050002 (576h42m3.684133047s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.065913082 (576h42m3.936996127s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.179797156 (576h42m4.050880221s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.231768326 (576h42m4.102851371s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456490982 (576h42m4.327574027s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456512182 (576h42m4.327595227s) [seg_pos={0 0}, upload=505569]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:11:04.456753241 (576h42m4.327836276s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.456760115 (576h42m4.32784316s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457005642 (576h42m4.328088687s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457012916 (576h42m4.328095961s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457082919 (576h42m4.328165964s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457120811 (576h42m4.328203846s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457152792 (576h42m4.328235837s) [seg_pos={0 0}, upload=505569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:11:04.457255488 (576h42m4.328338533s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:11:04.457262581 (576h42m4.328345616s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:11:04.748221661 (576h42m4.619304706s) [upload=505569]: Done uploading segments...
Dec 29 16:11:04.748245086 (576h42m4.619328131s) [upload=505569]: Upload finished. err: <nil>
Duration: 6.043343954s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.] duration:
3.80957686s trace:
https://jaeger.dev.storj.io/trace/57208fb4ada5351
FAIL (2025-12-29 16:13:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.809627387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
11.729402117s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 16:12:47 UTC)
orbiter warning: context deadline exceeded
Duration: 11.729444475s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
44.750822366s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 16:12:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 44.750836363s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X] duration:
5.001289239s trace:
https://jaeger.dev.storj.io/trace/57208fb4ada5351
FAIL (2025-12-29 16:12:36 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001311302s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..XX.XX...X.X..XX....XX.......X....X.X......X.....X......X.] duration:
6.458148353s trace:
https://jaeger.dev.storj.io/trace/35f0948d120e1a64
FAIL (2025-12-29 16:12:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 16:11:01.540027575 (576h42m1.41111063s) [upload=505569]: Starting upload
Dec 29 16:11:01.540125631 (576h42m1.411208677s) [upload=505569]: Done starting upload
Dec 29 16:11:01.540172460 (576h42m1.411255515s) [upload=505569]: Uploading segments...
Dec 29 16:11:01.540402268 (576h42m1.411485313s) [upload=505569]: Got next segment. Inline: false
Dec 29 16:11:01.540453786 (576h42m1.411536831s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 16:11:01.540459968 (576h42m1.411543013s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 16:11:01.796687083 (576h42m1.667770138s) [seg_pos={0 0}, upload=505569]: Begin upload segment...
Dec 29 16:11:01.796746415 (576h42m1.66782946s) [seg_pos={0 0}, upload=505569]: Done begin upload segment.
Dec 29 16:11:01.796782695 (576h42m1.66786574s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797236830 (576h42m1.668319885s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797759034 (576h42m1.668842079s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.797992658 (576h42m1.669075743s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798437265 (576h42m1.66952031s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.798865170 (576h42m1.669948215s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799287043 (576h42m1.670370098s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.799724426 (576h42m1.670807471s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800158061 (576h42m1.671241106s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.800580416 (576h42m1.671663461s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801347767 (576h42m1.672430813s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.801878428 (576h42m1.672961473s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802324087 (576h42m1.673407142s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.802762020 (576h42m1.673845065s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803196748 (576h42m1.674279793s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:01.803644912 (576h42m1.674727957s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Uploading piece...
Dec 29 16:11:03.406181393 (576h42m3.277264448s) [upload=505569]: Next returned nil segment
Dec 29 16:11:03.406199919 (576h42m3.277282964s) [upload=505569]: Waiting for error group managing segments...
Dec 29 16:11:03.790763095 (576h42m3.66184614s) [noise=true , address=145.79.227.4:4185 , piece_id=OD5RBJBXVFDCNH22RYT7KR4U47ZAYM5WRM5DBQMPOLXHVU4GGZZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.790929141 (576h42m3.662012186s) [noise=true , address=145.79.227.16:4912 , piece_id=O7K3Z2MARSJHFSWYKMZJCTB3BYKOVQBVY6HZXJFLM3F2EOUQJUNA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.796025666 (576h42m3.667108711s) [noise=true , address=145.79.227.38:4978 , piece_id=KE2M6D4P2QK5A6V6GQT4UV3DBHRSZPVP5XZHZMSJ4BGJ5KFOJTLA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.799086945 (576h42m3.67016999s) [noise=true , address=145.79.227.42:4122 , piece_id=XKX7ELMZ7MZERUM7I3PPO3MN6HJYED4VG7UPJW4OG4QNXCQJXUKQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.802427456 (576h42m3.673510501s) [noise=true , address=145.79.227.19:4077 , piece_id=27GF6LJA254VRRDI5V2HUCJTR6ELYDYEE2JUS4S2NQUGNYUTVWQQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.803905590 (576h42m3.674988635s) [noise=true , address=145.79.227.37:4566 , piece_id=PGYVAR6FTXYO2EYILSSURTY5LD43YWFB4SMUZHB2HRTJJU7QRWSA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.805330424 (576h42m3.676413469s) [noise=true , address=145.79.227.47:4265 , piece_id=4XK5PZR6PG3F63KSX3HUWQDIKIHRSKBH35V33J6TVGVUWZQOXWFA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.812366111 (576h42m3.683449156s) [noise=true , address=145.79.227.9:4366 , piece_id=DXLFO5GZEGJS52BR6EJUUSBVWX2E2GBNG56W373OEJLNQDOKNIPQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:03.813050002 (576h42m3.684133047s) [noise=true , address=145.79.227.14:4840 , piece_id=YSU6G5PHNLKRHLCJPILBFOHI7IKFJPP6MDPTWV3VVLJFPIN7HC7A, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.065913082 (576h42m3.936996127s) [noise=true , address=77.90.31.14:4664 , piece_id=2AQJ4S5MKBCKUSFO2R6PF4XCCS644Q3B7N7VPFEPKSS4SFBAJDAA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.179797156 (576h42m4.050880221s) [noise=true , address=45.91.116.7:4011 , piece_id=36HCH4PEXHHSC6B5OFSA3S2IU4D2UBEQ4YYIIE66BJNDI63OVAEQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.231768326 (576h42m4.102851371s) [noise=true , address=45.91.116.4:4937 , piece_id=VXDXDFZAKRD5E4BXFT5AAVJY7YB6PUIA2KAU5OHEJLWHTL4TKHZQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456490982 (576h42m4.327574027s) [noise=true , address=45.91.116.10:4082 , piece_id=F4F7SWKR2MLLHJKGVW3U42JV5HTR4VJGQ2NM3DUYQY4ZC3FHWDHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: <nil>
Dec 29 16:11:04.456512182 (576h42m4.327595227s) [seg_pos={0 0}, upload=505569]: Segment reached optimal threshold of 13 pieces.
Dec 29 16:11:04.456753241 (576h42m4.327836276s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.456760115 (576h42m4.32784316s) [noise=true , address=145.79.227.44:4193 , piece_id=XVFHAQ62WCWE73K3MFT73UGYIJ4SHUXBAD5RP3WVUO6LWLZ3JMHA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457005642 (576h42m4.328088687s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457012916 (576h42m4.328095961s) [noise=true , address=77.90.31.21:4880 , piece_id=ZSO7HRTRPYGHVQYXRIGVWOWZV5YB3RENBUNWVK7NWXZC34C6BEBQ, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457082919 (576h42m4.328165964s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 16:11:04.457120811 (576h42m4.328203846s) [noise=true , address=145.79.227.33:4869 , piece_id=NP2HH2RZNYH3STOOGJXT5K3T6LM7CGMJTMWHU6LOUI72UU3WV5AA, seg_pos={0 0}, upload=505569]: Long tail cancellation due to optimal threshold
Dec 29 16:11:04.457152792 (576h42m4.328235837s) [seg_pos={0 0}, upload=505569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 16:11:04.457255488 (576h42m4.328338533s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 16:11:04.457262581 (576h42m4.328345616s) [upload=505569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 16:11:04.748221661 (576h42m4.619304706s) [upload=505569]: Done uploading segments...
Dec 29 16:11:04.748245086 (576h42m4.619328131s) [upload=505569]: Upload finished. err: <nil>
Duration: 6.458167079s
core
uplink
us-select-1
|